Feed Supervisor.KnownTestDurations from the previous main run's ledger data

[jeremydmiller]

The follow-up queued since the supervised-CI work landed: Bobcat balances lanes longest-processing-time-first, but with no duration data it falls back to test count — measured on PersistenceTests as lanes finishing at 101.5s vs 11.4s, a quarter of the fleet idle. Every run's SupervisorResults already carried per-test durations; they were being thrown away.

The loop

1. Write (build/TestDurations.cs): every supervised run writes {Job}.{Project}.{Framework}.durations.json beside its ledger — first-attempt milliseconds per uid (a retry-amplified total would overweight exactly the flaky tests), unmeasured tests omitted rather than zero-filled. Rides the existing per-job test-ledger-* artifact upload untouched.
2. Publish: the flakiness roll-up copies every job's durations files verbatim into the test-ledger-run baseline artifact it already publishes from main (and excludes *.durations.json from its ledger-entry fold, which would otherwise trip its field assert). aggregate.json stays at the artifact root, so the existing baseline walk is unchanged.
3. Fetch (build/fetch-duration-baseline.sh): each test job downloads the last completed main run's baseline into previous-durations/ before ./build.sh runs — repo root, not artifacts/, because Clean wipes artifacts/ on every invocation. Same walk-a-few-runs-back logic as the flakiness baseline. The test job gains an explicit permissions: actions: read block because the download crosses runs.
4. Read: runSupervised feeds the matching file into Supervisor.KnownTestDurations.

Every hop is best-effort by design: no baseline (including the very first run after this merges), a stale one, or a renamed test degrades to exactly today's count balancing, with unknown tests charged the median of what is known — Bobcat's own rule.

Verified

• Run 1 (CoreTests, 2633 tests) wrote 2633 durations; seeded back into previous-durations/, run 2 logged balancing lanes with 2633 test duration(s) from the previous main run and passed 2633/2633.
• Uid stability across runs: 2633/2633 identical keys — xUnit v3's hashed uids are deterministic, so LPT balancing genuinely matches tests to their measured cost instead of silently falling through to medians.
• The roll-up exercised synthetically: durations excluded from entries.json (no assert trip), copied into the published durations/ directory.

Honest scope

Lane balancing only bites where MaxParallelWorkers > 1 — today that is CISqlServer (workers: 4) and local --test-workers runs. Everything else records the data for free, which is also the groundwork for duration trend reporting (Bobcat #56's committed-ledger layer).

🤖 Generated with Claude Code

[jeremydmiller]

The CISqlServer A/B, as promised

Two consecutive main runs of the same code — the only difference being whether a durations baseline existed. Control: run 32786336368 (first run post-merge, no baseline). Treatment: run 32791986856 (dispatched after the control published its baseline).

	control (count-balanced)	treatment (duration-balanced)
baseline fetch	none found … balance by count	run 32786336368, 44 job file(s)
balancing line	—	balancing lanes with 405 test duration(s)
lane plan	203 test(s) in 31 class(es), 202 test(s) in 31 class(es)	140 test(s) in 29 class(es), 265 test(s) in 33 class(es)
supervised wall clock	626s	669s
lane-finish spread (from heartbeats)	< ~30s — both lanes busy at 378/405	~30–60s — lane 1 done by 10m43s, lane 0 ran to 11m09s
result	405/405, 0 retries	405/405, 0 retries

The mechanism works — the treatment fetched the baseline, fed 405 durations in, and produced the deliberately uneven 140/265 plan that is LPT's signature.

The verdict is a no-win on this suite, and the durations data itself explains why: the baseline's total measured test time is 1370s, so the perfect-balance floor at 2 lanes is ~685s — and both runs sit essentially on it (626s and 669s, single samples on different runners). Count balancing already saturated both lanes to the final heartbeat; there was never imbalance here to recover. The 43s delta is within runner noise, and the 2026-08-01 profile this target's comment cites (931s total, ~470s LPT bound) describes a suite that has since grown to 405 post-filter tests and ~1370s.

What the loop is still worth: insurance for suites where counts don't align with cost — the motivating PersistenceTests measurement (count-balanced lanes at 101.5s vs 11.4s) is exactly the shape this fixes, and CISqlServer today just doesn't have it; plus the per-job durations corpus now accumulating on every main run, which is the input for duration trend reporting (Bobcat #56) and for re-checking this table when the suite's shape changes. Cost of the loop when it can't help: one artifact download and a dictionary.

Side-findings from the control run, for the record: its CIKafka job hit the 20-minute cap wedged on batch_processing_with_kafka.end_to_end, and the #4108 instrumentation named the test and pid in the log via heartbeats + STALLED: — but the partial ledger didn't write because the runner's SIGTERM never traverses the bash → build.sh → dotnet run chain. That gap is fixed by the pid-handshake relay in #4115.

🤖 Generated with Claude Code