Skip to content

fix(tests): serialize concurrent Herdr recoveries with the resume lock-wait opt-in - #72

Merged
arjmad merged 1 commit into
mainfrom
fm/fm-ci-herdr-behavior-flake
Oct 11, 2026
Merged

arjmad merged 1 commit into
mainfrom
fm/fm-ci-herdr-behavior-flake

Conversation

@arjmad

@arjmad arjmad commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Failure signature

The required Behavior tests (Herdr) job failed on the first attempt of PR #54 (run 37232993888, job 111526446200) and PR #58 (run 37241130813, job 111549911191), then passed on rerun.
Both failures are in tests/fm-backend-herdr-presentation-e2e.test.sh:

not ok - concurrent secondmate recovery failed: ...
error: herdr presentation recovery could not acquire its session lock; refusing a concurrent resume

The same signature failed two main pushes the same day (runs 37235749581 and 37184709629).
Across 118 Herdr-job attempts in this fork's last 100 CI runs (2026-09-23 to 2026-10-10), 5 failed, and 4 of those carry this signature.
The fifth was an unrelated Treehouse version mismatch on another branch.

Cause

The case "concurrent cross-home recoveries" stops the lab session, then starts two exact resumes at once (primary home and secondmate bravo) and asserts both succeed.
Both resumes ran on fm-spawn.sh's default resume path.
That path makes one bounded attempt at the session presentation-order lock (50 tries, 0.1s apart, about 5s total) and then hard-refuses by design.
Whichever resume loses the race succeeds only if the winner releases within that window.
On a loaded runner the winner's reclaim and launch handoff ran past 5s, so the loser refused.

The fm-spawn.sh header owns this contract: "A resume refuses by default on the same contention ... Pass --herdr-resume-lock-wait to opt that resume into waiting for the lock instead, so two concurrent recoveries can serialize and each still replace its own exact husk."
Upstream #6649 added that opt-in and its own deterministic refuse and wait cases, but left this concurrent case on the default path.
So the case asserted serialization while using the path that refuses contention.
Upstream main (1a86871, 2026-10-10) still has the same code.

Fix

Pass --herdr-resume-lock-wait to both concurrent resumes, the documented path for concurrent recoveries to serialize.
Bound each resume with the existing SPAWN_DEADLINE_SECONDS helper (180s), so a broken wait fails within a set time instead of hanging until the step timeout.
Every assertion stays: both resumes must succeed, each must replace its own exact husk in its original workspace, no old pane may remain, and focus must not drift.
No product code changes.
The refuse-by-default behavior is still covered by the existing deterministic case "default resumed identity refuses session lock contention", which holds the lock for 15s.
Waiting is still covered by "--herdr-resume-lock-wait waits out session lock contention instead of refusing", which holds it for 30s.

This is an upstream-owned test and the change applies to upstream unchanged.
It's the test-side fix for kunchenguid/firstmate#6454.
Upstream PR kunchenguid/firstmate#6496 instead changes the product default to a bounded 120s wait; #6649 later settled on refuse-by-default plus the opt-in, which this change follows.

Repeat evidence

A temporary probe workflow (run 38094826782, on a commit since dropped from this branch) ran the real-Herdr presentation e2e alone on 25 parallel GitHub runners with the pinned Herdr 0.7.4.
Two of the arms add CPU load, one busy loop per core, to reproduce the slow lock hold the failing runners hit.

Arm Test version CPU load Result
fixed this branch none 10/10 pass
fixed-load this branch busy loop per core 10/10 pass
unfixed-load base main busy loop per core 5/5 fail

All 20 fixed runs printed ok - real Herdr lab: concurrent cross-home recoveries replace exact husks under one session lock with no focus drift and no not ok line.
All 5 unfixed runs failed with the same error as CI: herdr presentation recovery could not acquire its session lock; refusing a concurrent resume.
Bravo lost the race in four of them and the primary lost it in one, so either recovery can be the one that refuses.

bin/fm-lint.sh tests/fm-backend-herdr-presentation-e2e.test.sh passes with the pinned ShellCheck 0.11.0.

…k-wait opt-in

The real-Herdr presentation e2e case "concurrent cross-home recoveries"
starts two exact resumes at once and asserts both succeed, but it ran them
on the default resume path, which makes one bounded ~5s attempt at the
session presentation-order lock and then hard-refuses. When the first
recovery held the lock past that bound on a loaded runner, the second
failed with "herdr presentation recovery could not acquire its session
lock; refusing a concurrent resume".

fm-spawn.sh documents --herdr-resume-lock-wait as the path for two
concurrent recoveries to serialize, so pass it to both resumes and bound
each with the existing SPAWN_DEADLINE_SECONDS deadline. The default
refusal stays covered by its own deterministic contention case.
@arjmad
arjmad force-pushed the fm/fm-ci-herdr-behavior-flake branch from fad093c to 251c395 Compare October 10, 2026 23:58
@arjmad
arjmad marked this pull request as ready for review October 10, 2026 23:58
@arjmad
arjmad merged commit 71184b2 into main Oct 11, 2026
47 of 48 checks passed
@arjmad
arjmad deleted the fm/fm-ci-herdr-behavior-flake branch October 11, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant