Repository navigation
fix(tests): serialize concurrent Herdr recoveries with the resume lock-wait opt-in - #72
Merged
Merged
Conversation
…k-wait opt-in The real-Herdr presentation e2e case "concurrent cross-home recoveries" starts two exact resumes at once and asserts both succeed, but it ran them on the default resume path, which makes one bounded ~5s attempt at the session presentation-order lock and then hard-refuses. When the first recovery held the lock past that bound on a loaded runner, the second failed with "herdr presentation recovery could not acquire its session lock; refusing a concurrent resume". fm-spawn.sh documents --herdr-resume-lock-wait as the path for two concurrent recoveries to serialize, so pass it to both resumes and bound each with the existing SPAWN_DEADLINE_SECONDS deadline. The default refusal stays covered by its own deterministic contention case.
arjmad
force-pushed
the
fm/fm-ci-herdr-behavior-flake
branch
from
October 10, 2026 23:58
fad093c to
251c395
Compare
arjmad
marked this pull request as ready for review
October 10, 2026 23:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failure signature
The required
Behavior tests (Herdr)job failed on the first attempt of PR #54 (run 37232993888, job 111526446200) and PR #58 (run 37241130813, job 111549911191), then passed on rerun.Both failures are in
tests/fm-backend-herdr-presentation-e2e.test.sh:The same signature failed two
mainpushes the same day (runs 37235749581 and 37184709629).Across 118 Herdr-job attempts in this fork's last 100 CI runs (2026-09-23 to 2026-10-10), 5 failed, and 4 of those carry this signature.
The fifth was an unrelated Treehouse version mismatch on another branch.
Cause
The case "concurrent cross-home recoveries" stops the lab session, then starts two exact resumes at once (primary home and secondmate bravo) and asserts both succeed.
Both resumes ran on
fm-spawn.sh's default resume path.That path makes one bounded attempt at the session presentation-order lock (50 tries, 0.1s apart, about 5s total) and then hard-refuses by design.
Whichever resume loses the race succeeds only if the winner releases within that window.
On a loaded runner the winner's reclaim and launch handoff ran past 5s, so the loser refused.
The
fm-spawn.shheader owns this contract: "A resume refuses by default on the same contention ... Pass --herdr-resume-lock-wait to opt that resume into waiting for the lock instead, so two concurrent recoveries can serialize and each still replace its own exact husk."Upstream #6649 added that opt-in and its own deterministic refuse and wait cases, but left this concurrent case on the default path.
So the case asserted serialization while using the path that refuses contention.
Upstream main (1a86871, 2026-10-10) still has the same code.
Fix
Pass
--herdr-resume-lock-waitto both concurrent resumes, the documented path for concurrent recoveries to serialize.Bound each resume with the existing
SPAWN_DEADLINE_SECONDShelper (180s), so a broken wait fails within a set time instead of hanging until the step timeout.Every assertion stays: both resumes must succeed, each must replace its own exact husk in its original workspace, no old pane may remain, and focus must not drift.
No product code changes.
The refuse-by-default behavior is still covered by the existing deterministic case "default resumed identity refuses session lock contention", which holds the lock for 15s.
Waiting is still covered by "--herdr-resume-lock-wait waits out session lock contention instead of refusing", which holds it for 30s.
This is an upstream-owned test and the change applies to upstream unchanged.
It's the test-side fix for kunchenguid/firstmate#6454.
Upstream PR kunchenguid/firstmate#6496 instead changes the product default to a bounded 120s wait; #6649 later settled on refuse-by-default plus the opt-in, which this change follows.
Repeat evidence
A temporary probe workflow (run 38094826782, on a commit since dropped from this branch) ran the real-Herdr presentation e2e alone on 25 parallel GitHub runners with the pinned Herdr 0.7.4.
Two of the arms add CPU load, one busy loop per core, to reproduce the slow lock hold the failing runners hit.
mainAll 20 fixed runs printed
ok - real Herdr lab: concurrent cross-home recoveries replace exact husks under one session lock with no focus driftand nonot okline.All 5 unfixed runs failed with the same error as CI:
herdr presentation recovery could not acquire its session lock; refusing a concurrent resume.Bravo lost the race in four of them and the primary lost it in one, so either recovery can be the one that refuses.
bin/fm-lint.sh tests/fm-backend-herdr-presentation-e2e.test.shpasses with the pinned ShellCheck 0.11.0.