Repository navigation
test: split the supervision host suite so serial shards can spread it - #7031
Marsjohn-11 wants to merge 1 commit into
Conversation
tests/fm-supervision-host.test.sh took 930-1743 s on ubuntu-latest and alone carried portable serial 2 to its 30-minute cap. Split its 75 cases, unchanged, into six suites over a shared tests/supervision-host-helpers.sh fixture: the core report, dispatch, drain, and attended host cases, plus -stop-hook, -mirror, -away, -latch, and -park siblings. Give each a serial packing hint from the slowest per-case CI samples, register the siblings in the afk family and the Pi extension change map, and update the docs that name the suite. Inventory, before (one file) and after (six files plus helpers): - test cases defined: 75 -> 75, same names; runner calls: 75 -> 75 - assertion calls (assert_*, expect_code, fail, pass) in test functions: 848 -> 848 - every test and helper function (120) is byte-identical to the original Per suite: cases, assertion calls, packing hint - fm-supervision-host.test.sh 33 367 232874 ms - fm-supervision-host-stop-hook.test.sh 15 174 153674 ms - fm-supervision-host-mirror.test.sh 5 86 398774 ms - fm-supervision-host-away.test.sh 11 133 304574 ms - fm-supervision-host-latch.test.sh 3 69 441474 ms - fm-supervision-host-park.test.sh 8 80 274774 ms Hints sum each case's slowest duration across main runs 37699232582, 37720051397, 37739997864, 37829693397, 37973072108, 38090672222 and PR runs 38090029535, 38090723262 (from FM_TEST_BEGIN and consecutive ok-line timestamps), plus the largest observed 11274 ms tail to FM_TEST_END. Planner estimates (bin/fm-test-run.sh, 9 serial shards): every shard packs 1119472-1119504 ms (~18.7 min) against the 20-minute packing target and the unchanged 30-minute job cap; the largest single suite drops from 1743 s to ~441 s. Shard 1 is the fm-watch-triage.test.sh floor (1074843 ms).
|
Speaking as Kun's firstmate: thank you @Marsjohn-11 — this is a careful split. I checked it mechanically against main VISION (compact): all rules align or are neutral (test-only; "field incidents become regression coverage" and repo discipline served by keeping serial shards under the cap). Contract-class: restore — no product behavior changes; it puts the serial shards back under the documented packing target that the single file (which also times out serial 2 on main) was breaking. Not waiting on you. One ordering question goes to Kun: open PR #6753 (already with Kun for a decision) splits this same file a different way ( |
Before/after inventory
tests/supervision-host-helpers.sh)assert_*,expect_code,fail,pass) in test functionsfm-supervision-host.test.shfm-supervision-host-stop-hook.test.shfm-supervision-host-mirror.test.shfm-supervision-host-away.test.shfm-supervision-host-latch.test.shfm-supervision-host-park.test.shThe planner models every one of the 9 serial shards at about 1129.5 s (18.8 min) against the 20-minute packing target and the unchanged 30-minute job cap.
On this PR's first CI round all six suites passed in about 431 s combined, against 930-1743 s for the single file on recent runs, and the slowest serial shard took 22m59s (serial 6, which carries none of these suites).
The hints are conservative per-case maxima from eight earlier CI runs; refresh them from the split suites' own timing artifacts per
docs/fm-test-portable-shards.md.What Changed
tests/fm-supervision-host.test.shinto that suite plus five siblings:-stop-hook,-mirror,-away,-latch, and-park. The siblings share fixtures through the newtests/supervision-host-helpers.sh.bin/fm-test-run.shso all six suites map to theafkfamily and are selected together when supervision host paths change. The script also has new per-suite duration hints, derived from the per-case CI timings of the old combined suite, which replace its single 789123 ms hint.docs/fm-test-portable-shards.md,docs/supervision-host.md, anddocs/verification/supervision.mdto list the new suites and explain where the new duration hints come from.Risk Assessment
✅ Low: A mechanical comparison found the split faithful: all 120 functions are byte-identical to the original, all 75 runner calls are kept in their original relative order, and the top-level setup is intact. No suite uses a function defined in another suite. The new suites stay in the non-concurrent afk family, so they still run serially, and the runner change and hint edits are consistent.
Testing
I drove the real test runner. Coverage passes and the planner balances all nine serial shards at a modeled 18.8 min. At base, the single fm-supervision-host.test.sh sat on shard 2 under an optimistic 789s hint, and in CI it actually ran 1100-1743s. Its six replacement suites now land on five different shards, and the largest single hint is 441s. The 75-case set is identical. The CI 30-minute policy guard passes and .github is untouched. The afk family and changed-path selection pick up all six suites. The core, latch and park suites passed under the runner; latch's pass was a solo rerun. To save time I first ran the suites concurrently. That is unsupported: the afk family is not proven concurrent-safe, and CI runs these serially. Five suites hit timing failures that way. Latch then passed when rerun alone, which points to my concurrent setup rather than the split. Stop-hook, mirror and away were never rerun alone before the time budget ran out, so they are reported untested. There is no UI surface, so there are no screenshots.
bin/fm-test-run.sh tests/fm-supervision-host-latch.test.sh: 3 ok, exit=0, 327sEvidence: Modeled serial-shard weights, base vs target
Source: Modeled serial-shard weights, base vs target
Evidence: Case inventory: 75 case calls, identical before and after
Source: Case inventory: 75 case calls, identical before and after
Evidence: afk family and changed-path selection
Source: afk family and changed-path selection
Evidence: Core supervision-host suite run (exit 0)
Source: Core supervision-host suite run (exit 0)
Evidence: Park suite run (exit 0)
Source: Park suite run (exit 0)
Evidence: CI workflow timeout-policy test (exit 0)
Source: CI workflow timeout-policy test (exit 0)
~/.no-mistakes/evidence/01M4MK3B0YVB3EHWHRBRQVGRJY/concurrent-6way-invalid)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
CONTRIBUTING.md:138- CONTRIBUTING.md:138 lists the shared test helpers (tests/lib.sh, fixtures.sh, wake-helpers.sh, secondmate-helpers.sh, git-config-helpers.sh), but it does not list the new tests/supervision-host-helpers.sh. A contributor reading that list would not learn that the supervision host fixture now lives there. Adding it to the list fixes this.✅ **Test** - passed
✅ No issues found.
bin/fm-test-run.sh tests/fm-supervision-host-latch.test.sh: 3 ok, exit=0, 327sbin/fm-test-run.sh --check-coverageat the target commit and at base 4735ddc (base snapshot extracted to a throwaway temp dir)bin/fm-test-run.sh --list --lane portable-serial-<k>of9for k=1..9, base vs target, with each shard's weight summed from the serial hint tabletest-case calls in base tests/fm-supervision-host.test.sh compared against the union of the six new suitesbin/fm-test-run.sh --list --family afkand--list --changed --base 4735ddcbash tests/fm-ci-workflow.test.sh(30-minute Normal-tier policy guard); confirmed .github is unchangedbin/fm-test-run.sh tests/fm-supervision-host.test.sh(exit 0, 720s)bin/fm-test-run.sh tests/fm-supervision-host-latch.test.shrun alone (exit 0, 327s)bin/fm-test-run.sh tests/fm-supervision-host-park.test.sh(exit 0, 221s)stop-hook, mirror and away suites, run only concurrently (6-way, then 2 chains): they failed on timing and these results were set aside as invalid✅ **Document** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.