Retry ledger: record WHY a flaky test failed, not just which one (GH-3787)

[jeremydmiller]

The ledger named the test that only passed on a retry and threw away the reason. Bobcat's log keeps only the [FLAKY] ... passed on attempt 2 line for a test that eventually passed — the first attempt's failure appears nowhere at all. That is the difference between a lead and a name.

It is currently load-bearing. Of the four flaky tests standing on main on 2026-08-03, two were undiagnosable for exactly this reason, and the note already sitting on multi_tenancy_through_virtual_hosts records its own next step as:

Next step is to dump the tracked session on the FIRST attempt rather than infer from the assertion.

Which is this.

The change

Each retried test now carries the failing attempt's ErrorType and message into the ledger JSON, the job step summary, and the cross-job roll-up:

- **CIOracle**
  - `OracleTests.Transport.clear_all_wolverine_storage.is_idempotent`
    - attempt 1: Shouldly.ShouldAssertException — counts.Queued should be 1L but was 2L

One trap worth recording

The reason comes from SupervisorAttempt.Outcome — the test's own error — and deliberately not from Disposition.Reason. The latter is the supervisor's rationale for retrying:

a failure is retried in a fresh process, within the budget, to separate flaky from broken

…which is the same sentence for every retry in the repository. The first version of this change read that field. It compiled, serialized and rendered perfectly, and carried no information whatsoever. Only forcing a real retry and reading the output showed the difference — a green run could never have caught it.

Backwards compatibility

FlakyTests stays a plain string array. flakiness-report.sh flattens it with map(.FlakyTests[]), and the baseline it diffs against is downloaded from an earlier run's artifact, so the new field is additive and read through // [] — without that default a single pre-change baseline would null the whole roll-up.

Verification

End to end with a test rigged to fail its first attempt and pass on the retry:

"FlakyFailures": [
  {
    "Test": "SqliteTests.TempFlakyProbe.fails_once_then_passes",
    "Attempt": 1,
    "ErrorType": "System.InvalidOperationException",
    "Reason": "System.InvalidOperationException : Deliberate first-attempt failure for ledger probe"
  }
]

Both the step summary and the roll-up render it under the test name. Aggregation and rendering were also checked against a mixed set containing a pre-change ledger, which degrades to the name alone rather than failing. bash -n passes on the script.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m