Skip to content

test: split the supervision host suite so serial shards can spread it - #7031

Open
Marsjohn-11 wants to merge 1 commit into
kunchenguid:mainfrom
Marsjohn-11:fix/supervision-host-test-split
Open

Marsjohn-11 wants to merge 1 commit into
kunchenguid:mainfrom
Marsjohn-11:fix/supervision-host-test-split

Conversation

@Marsjohn-11

@Marsjohn-11 Marsjohn-11 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Before/after inventory

Before (one file) After (six suites + tests/supervision-host-helpers.sh)
Test cases defined 75 75 (same names)
Runner calls 75 75 (original relative order)
Assertion calls (assert_*, expect_code, fail, pass) in test functions 848 848
Test and helper functions byte-identical to the original - 120 of 120
Suite Cases Assertion calls Packing hint CI on this PR
fm-supervision-host.test.sh 33 367 232874 ms 91124 ms (serial 8)
fm-supervision-host-stop-hook.test.sh 15 174 153674 ms 83150 ms (serial 7)
fm-supervision-host-mirror.test.sh 5 86 398774 ms 112533 ms (serial 3)
fm-supervision-host-away.test.sh 11 133 304574 ms 52156 ms (serial 7)
fm-supervision-host-latch.test.sh 3 69 441474 ms 53681 ms (serial 2)
fm-supervision-host-park.test.sh 8 80 274774 ms 37914 ms (serial 9)

The planner models every one of the 9 serial shards at about 1129.5 s (18.8 min) against the 20-minute packing target and the unchanged 30-minute job cap.
On this PR's first CI round all six suites passed in about 431 s combined, against 930-1743 s for the single file on recent runs, and the slowest serial shard took 22m59s (serial 6, which carries none of these suites).
The hints are conservative per-case maxima from eight earlier CI runs; refresh them from the split suites' own timing artifacts per docs/fm-test-portable-shards.md.

What Changed

  • Split the single tests/fm-supervision-host.test.sh into that suite plus five siblings: -stop-hook, -mirror, -away, -latch, and -park. The siblings share fixtures through the new tests/supervision-host-helpers.sh.
  • Updated bin/fm-test-run.sh so all six suites map to the afk family and are selected together when supervision host paths change. The script also has new per-suite duration hints, derived from the per-case CI timings of the old combined suite, which replace its single 789123 ms hint.
  • Updated docs/fm-test-portable-shards.md, docs/supervision-host.md, and docs/verification/supervision.md to list the new suites and explain where the new duration hints come from.

Risk Assessment

✅ Low: A mechanical comparison found the split faithful: all 120 functions are byte-identical to the original, all 75 runner calls are kept in their original relative order, and the top-level setup is intact. No suite uses a function defined in another suite. The new suites stay in the non-concurrent afk family, so they still run serially, and the runner change and hint edits are consistent.

Testing

I drove the real test runner. Coverage passes and the planner balances all nine serial shards at a modeled 18.8 min. At base, the single fm-supervision-host.test.sh sat on shard 2 under an optimistic 789s hint, and in CI it actually ran 1100-1743s. Its six replacement suites now land on five different shards, and the largest single hint is 441s. The 75-case set is identical. The CI 30-minute policy guard passes and .github is untouched. The afk family and changed-path selection pick up all six suites. The core, latch and park suites passed under the runner; latch's pass was a solo rerun. To save time I first ran the suites concurrently. That is unsupported: the afk family is not proven concurrent-safe, and CI runs these serially. Five suites hit timing failures that way. Latch then passed when rerun alone, which points to my concurrent setup rather than the split. Stop-hook, mirror and away were never rerun alone before the time budget ran out, so they are reported untested. There is no UI surface, so there are no screenshots.

  • Live validation: ✅ go - 7 of 8 scenarios driven live against the product
Scenario Result Live Evidence
The shard planner spreads the split suites across serial lanes, and every modeled shard fits under the 20-min packing target ✅ pass live serial-shard-model.txt: the suites land on shards 2, 3, 7, 8 and 9, and all 9 shards model at about 1129.5s (18.8 min) against a 1200s budget. --check-coverage reports ok with serial_max_ms=1129498.
Every one of the 75 original cases is still run after the split, with no duplicates ✅ pass live case-inventory.txt: base 75 calls, target 75 calls, 75 unique, and the sorted sets are identical
The 30-minute Normal-tier CI bound is not raised and its guard still passes ✅ pass live fm-ci-workflow.log exit=0; git diff shows .github unchanged
Selecting the afk family or changed paths picks up all six suites ✅ pass live family-selection.txt
Core fm-supervision-host.test.sh passes under the runner ✅ pass live fm-supervision-host.log exit=0, duration 720s
fm-supervision-host-latch.test.sh passes when run alone ✅ pass live solo bin/fm-test-run.sh tests/fm-supervision-host-latch.test.sh: 3 ok, exit=0, 327s
fm-supervision-host-park.test.sh passes under the runner ✅ pass live fm-supervision-host-park.log exit=0, 221s
fm-supervision-host-stop-hook, -mirror and -away suites pass when run serially as CI does ⏸️ untested no These suites are not proven concurrent-safe, and CI runs them serially, so my concurrent attempts do not count as results. Latch failed the same way under concurrency and then passed alone, so the fai…
Evidence: Modeled serial-shard weights, base vs target

Source: Modeled serial-shard weights, base vs target

\### base 4735ddc
serial 1of9:   1 scripts, modeled 1074843 ms (17.9 min)
serial 2of9:  21 scripts, modeled 1009192 ms (16.8 min)
serial 3of9:  27 scripts, modeled 1009221 ms (16.8 min)
serial 4of9:  26 scripts, modeled 1009189 ms (16.8 min)
serial 5of9:  28 scripts, modeled 1009210 ms (16.8 min)
serial 6of9:  27 scripts, modeled 1009191 ms (16.8 min)
serial 7of9:  26 scripts, modeled 1009188 ms (16.8 min)
serial 8of9:  27 scripts, modeled 1009190 ms (16.8 min)
serial 9of9:  28 scripts, modeled 1009188 ms (16.8 min)
\### target 6d4d61b
serial 1of9:  13 scripts, modeled 1129498 ms (18.8 min)
serial 2of9:  26 scripts, modeled 1129490 ms (18.8 min)
serial 3of9:  26 scripts, modeled 1129496 ms (18.8 min)
serial 4of9:  25 scripts, modeled 1129498 ms (18.8 min)
serial 5of9:  25 scripts, modeled 1129489 ms (18.8 min)
serial 6of9:  26 scripts, modeled 1129490 ms (18.8 min)
serial 7of9:  26 scripts, modeled 1129487 ms (18.8 min)
serial 8of9:  23 scripts, modeled 1129487 ms (18.8 min)
serial 9of9:  26 scripts, modeled 1129498 ms (18.8 min)
Evidence: Case inventory: 75 case calls, identical before and after

Source: Case inventory: 75 case calls, identical before and after

runner calls base=75 target=75 uniq_target=75
identical case-call sets
Evidence: afk family and changed-path selection

Source: afk family and changed-path selection

$ fm-test-run.sh --list --family afk
tests/fm-afk-contract.test.sh
tests/fm-afk-inject-e2e.test.sh
tests/fm-afk-return.test.sh
tests/fm-host-mirror.test.sh
tests/fm-supervision-host-away.test.sh
tests/fm-supervision-host-latch.test.sh
tests/fm-supervision-host-mirror.test.sh
tests/fm-supervision-host-park.test.sh
tests/fm-supervision-host-stop-hook.test.sh
tests/fm-supervision-host.test.sh
$ fm-test-run.sh --list --changed --base 4735ddc
tests/fm-supervision-host-away.test.sh
tests/fm-supervision-host-latch.test.sh
tests/fm-supervision-host-mirror.test.sh
tests/fm-supervision-host-park.test.sh
tests/fm-supervision-host-stop-hook.test.sh
tests/fm-supervision-host.test.sh
Evidence: Core supervision-host suite run (exit 0)

Source: Core supervision-host suite run (exit 0)

FM_TEST_BEGIN 2026-10-11T04:24:06Z tests/fm-supervision-host.test.sh family=afk expected_gate_skip=none
ok - report surface: only the branch actor's current turn may report, and only on the tasks its wake names
ok - report surface: visible late outcomes queue a relay, while silent outcomes remain stored without a wake or note
ok - dispatch entry: the host reads branch eligibility, the offer rule, and the wake prompt from the Pi branch's own owner
ok - drain: BRANCH OUTCOMES runs on a Claude home by default and on another primary with the file, never with off, and never on Pi
ok - drain: captain outcomes come first, and routine overflow collapses into a count one drain clears
ok - drain: repeated captain outcomes collapse per task, and the byte cap presents only the run its acknowledgement covers
ok - drain: a long away window costs one short drain, captain outcomes collapsed per task and routine overflow counted, and nothing from it is shown again
ok - drain: the BRANCH OUTCOMES budgets count bytes, cutting multibyte summaries by whole characters in any locale
ok - drain: branch outcomes stay unread when a projection of the store fails
ok - drain: branch outcomes stay unread and the drain fails when jq is missing
ok - drain: branch outcomes stay unread when the drain cannot print them
ok - drain: a legacy backlog is presented with each outcome's age and a check-first instruction, never adopted
ok - drain: a keyed decision survives acknowledgement through a newer outcome for its task
ok - drain: an outcome carried across a switch off Pi comes back with its age, never adopted
ok - drain: an outcome nothing has shown is presented until acknowledged, and a repeated acknowledgement changes nothing
ok - drain: an outcome the host drain presented but main never acknowledged stays unprocessed across a switch to Pi
ok - drain: an outcome the host drain presented but main never acknowledged survives an outcome index repair
ok - host: an attended wake the branch may take is handled on the engine, and its routine outcome never wakes main
ok - host: an attended captain outcome wakes main once and stays in its drain until main acknowledges it
ok - host: a captain outcome recorded after the captain left waits for the return, then reaches main's drain
ok - host: a quiet record without its daemon is a present captain, so outcomes and decisions reach main
ok - host: an attended decision close stays main's exactly as the plain arm delivers it
ok - host: an off written while the host is parked sends the next attended close to main, naming the opt-out
ok - host: a main-only pass-through leaves the successor watcher running and the close undelivered for main
ok - host: an attended close whose main session cannot be identified reaches main and runs no engine turn
ok - host: a decision close accepted away whose turn starts attended still reaches main unchanged
ok - host: an attended close whose task turns main-only before its turn still reaches main unchanged
ok - host: an engine turn is bounded, and tool processes outside its process group are reaped
ok - host: a restarted host stops, by recorded identity, the cycle a killed predecessor left running
ok - host: an unchanged held outcome reaches the captain once across cadences, and a later decision on the task still does
ok - host: a home naming an unverified engine hands every away wake to main with the reason
ok - host: a host outside the session-lock owner stands down without arming
ok - host: a host under a superseded auto-arm generation stands down without touching the owner
FM_TEST_END 2026-10-11T04:36:06Z tests/fm-supervision-host.test.sh exit=0 duration_ms=720348 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=720723
FM_TEST_SUMMARY_FAMILY family=afk count=1 duration_ms=720348 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-supervision-host.test.sh duration_ms=720348
fm-test-run: wrote timing artifact: ~/.no-mistakes/evidence/01M4MK3B0YVB3EHWHRBRQVGRJY/fm-supervision-host.json
exit=0 secs=721
Evidence: Park suite run (exit 0)

Source: Park suite run (exit 0)

FM_TEST_BEGIN 2026-10-11T04:41:06Z tests/fm-supervision-host-park.test.sh family=afk expected_gate_skip=none
ok - host: parked child-exit sampling uses ordinary half-second sleeps
ok - host: the park ends itself with a boundary wake and a stopped watcher
ok - host: waiting closes cannot carry the park past its boundary
ok - host: a close whose margin runs out while the successor starts reaches main at the boundary without a turn
ok - host: the park's test clock is inert without the test marker
ok - host: a park at or beyond the Stop-hook registration falls back to the default boundary
ok - host: an owner's later park limit lets a turn outlive the boundary, and no other limit does
ok - host: the first cycle's status streams once, --restart replaces a stale watcher, and an owner predecessor makes a handling successor
FM_TEST_END 2026-10-11T04:44:47Z tests/fm-supervision-host-park.test.sh exit=0 duration_ms=220511 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=220705
FM_TEST_SUMMARY_FAMILY family=afk count=1 duration_ms=220511 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-supervision-host-park.test.sh duration_ms=220511
fm-test-run: wrote timing artifact: ~/.no-mistakes/evidence/01M4MK3B0YVB3EHWHRBRQVGRJY/fm-supervision-host-park.json
exit=0
Evidence: CI workflow timeout-policy test (exit 0)

Source: CI workflow timeout-policy test (exit 0)

ok - CI matrices cover every executable serial lane and canonical lint root exactly once
ok - a newer push to one PR supersedes that PR's in-flight CI
ok - distinct PRs get distinct concurrency groups
ok - every main push keeps its own group and is never cancelled
ok - every ci.yml job carries a finite timeout
ok - every ci.yml job belongs to one of the three timeout tiers
ok - fast tier jobs share one 5 minute tripwire
ok - normal tier jobs share one 30 minute budget
ok - Herdr keeps a 20 minute step tripwire under a 75 minute job backstop
exit=0
  • Evidence: Invalid 6-way concurrent attempt (not a valid signal) (local file: ~/.no-mistakes/evidence/01M4MK3B0YVB3EHWHRBRQVGRJY/concurrent-6way-invalid)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ℹ️ CONTRIBUTING.md:138 - CONTRIBUTING.md:138 lists the shared test helpers (tests/lib.sh, fixtures.sh, wake-helpers.sh, secondmate-helpers.sh, git-config-helpers.sh), but it does not list the new tests/supervision-host-helpers.sh. A contributor reading that list would not learn that the supervision host fixture now lives there. Adding it to the list fixes this.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 7 of 8 scenarios driven live against the product
Scenario Result Live Evidence
The shard planner spreads the split suites across serial lanes, and every modeled shard fits under the 20-min packing target ✅ pass live serial-shard-model.txt: the suites land on shards 2, 3, 7, 8 and 9, and all 9 shards model at about 1129.5s (18.8 min) against a 1200s budget. --check-coverage reports ok with serial_max_ms=1129498.
Every one of the 75 original cases is still run after the split, with no duplicates ✅ pass live case-inventory.txt: base 75 calls, target 75 calls, 75 unique, and the sorted sets are identical
The 30-minute Normal-tier CI bound is not raised and its guard still passes ✅ pass live fm-ci-workflow.log exit=0; git diff shows .github unchanged
Selecting the afk family or changed paths picks up all six suites ✅ pass live family-selection.txt
Core fm-supervision-host.test.sh passes under the runner ✅ pass live fm-supervision-host.log exit=0, duration 720s
fm-supervision-host-latch.test.sh passes when run alone ✅ pass live solo bin/fm-test-run.sh tests/fm-supervision-host-latch.test.sh: 3 ok, exit=0, 327s
fm-supervision-host-park.test.sh passes under the runner ✅ pass live fm-supervision-host-park.log exit=0, 221s
fm-supervision-host-stop-hook, -mirror and -away suites pass when run serially as CI does ⏸️ untested no These suites are not proven concurrent-safe, and CI runs them serially, so my concurrent attempts do not count as results. Latch failed the same way under concurrency and then passed alone, so the fai…
  • bin/fm-test-run.sh --check-coverage at the target commit and at base 4735ddc (base snapshot extracted to a throwaway temp dir)
  • bin/fm-test-run.sh --list --lane portable-serial-<k>of9 for k=1..9, base vs target, with each shard's weight summed from the serial hint table
  • test-case calls in base tests/fm-supervision-host.test.sh compared against the union of the six new suites
  • bin/fm-test-run.sh --list --family afk and --list --changed --base 4735ddc
  • bash tests/fm-ci-workflow.test.sh (30-minute Normal-tier policy guard); confirmed .github is unchanged
  • bin/fm-test-run.sh tests/fm-supervision-host.test.sh (exit 0, 720s)
  • bin/fm-test-run.sh tests/fm-supervision-host-latch.test.sh run alone (exit 0, 327s)
  • bin/fm-test-run.sh tests/fm-supervision-host-park.test.sh (exit 0, 221s)
  • stop-hook, mirror and away suites, run only concurrently (6-way, then 2 chains): they failed on timing and these results were set aside as invalid
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

tests/fm-supervision-host.test.sh took 930-1743 s on ubuntu-latest and
alone carried portable serial 2 to its 30-minute cap. Split its 75 cases,
unchanged, into six suites over a shared tests/supervision-host-helpers.sh
fixture: the core report, dispatch, drain, and attended host cases, plus
-stop-hook, -mirror, -away, -latch, and -park siblings. Give each a
serial packing hint from the slowest per-case CI samples, register the
siblings in the afk family and the Pi extension change map, and update
the docs that name the suite.

Inventory, before (one file) and after (six files plus helpers):
- test cases defined: 75 -> 75, same names; runner calls: 75 -> 75
- assertion calls (assert_*, expect_code, fail, pass) in test
  functions: 848 -> 848
- every test and helper function (120) is byte-identical to the original

Per suite: cases, assertion calls, packing hint
- fm-supervision-host.test.sh            33  367  232874 ms
- fm-supervision-host-stop-hook.test.sh  15  174  153674 ms
- fm-supervision-host-mirror.test.sh      5   86  398774 ms
- fm-supervision-host-away.test.sh       11  133  304574 ms
- fm-supervision-host-latch.test.sh       3   69  441474 ms
- fm-supervision-host-park.test.sh        8   80  274774 ms

Hints sum each case's slowest duration across main runs 37699232582,
37720051397, 37739997864, 37829693397, 37973072108, 38090672222 and PR
runs 38090029535, 38090723262 (from FM_TEST_BEGIN and consecutive ok-line
timestamps), plus the largest observed 11274 ms tail to FM_TEST_END.

Planner estimates (bin/fm-test-run.sh, 9 serial shards): every shard
packs 1119472-1119504 ms (~18.7 min) against the 20-minute packing target
and the unchanged 30-minute job cap; the largest single suite drops from
1743 s to ~441 s. Shard 1 is the fm-watch-triage.test.sh floor (1074843 ms).
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thank you @Marsjohn-11 — this is a careful split. I checked it mechanically against main 4735ddc6: all 109 function definitions across the six suites + tests/supervision-host-helpers.sh are byte-identical to the original file, no duplicates or extras, and the 75 runner calls are the same set. The bin/fm-test-run.sh change is family membership, changed-path selection, and duration hints only; no .github, network, or secret changes. CI is fully green on 6d4d61b3, the latest "PR must be raised via no-mistakes" run passed, the attestation matches this head, and the merge state is CLEAN.

VISION (compact): all rules align or are neutral (test-only; "field incidents become regression coverage" and repo discipline served by keeping serial shards under the cap). Contract-class: restore — no product behavior changes; it puts the serial shards back under the documented packing target that the single file (which also times out serial 2 on main) was breaking.

Not waiting on you. One ordering question goes to Kun: open PR #6753 (already with Kun for a decision) splits this same file a different way (fm-supervision-host-late.test.sh), so whichever lands first makes the other conflict. I'm holding this briefly for that call instead of landing it out from under #6753.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants