CI retry counts are invisible: a job burning 22 of 25 retries is indistinguishable from a clean green

[jeremydmiller]

CIAzureServiceBus reported green for four consecutive main runs while burning 22 of its
25-retry budget on every one of them — the same 22 tests, failing on first attempt and passing
alone in a fresh process. That was 85% of all flakiness in the repository, and it was invisible.

Nothing in the GitHub checks UI distinguishes a job at 22/25 retries from one at 0/25. Both are a
green tick.

What exists today, precisely

build/SupervisedTests.cs logs, via Serilog:

=== Wolverine.AzureServiceBus.Tests: 239 passed (0 retries, 2 worker processes) ===
  [FLAKY] <test> — passed on attempt 2

Nuke's GitHub Actions sink does turn those warnings into real ::warning annotations, so the
Annotations panel is not empty. It is worse than empty — it is truncated at exactly the number
that carries the signal. On run 30759855844, the job that spent 22 retries:

$ gh api ".../actions/jobs/91528323381/logs" | grep -c FLAKY
66                       # 22 flaky tests, logged three ways

$ gh api ".../check-runs/91528323381/annotations?per_page=100" --paginate -q length
10                       # GitHub caps warning annotations at 10 per check run, first-wins

So the panel showed ten [FLAKY] lines. It would have shown ten at 10 retries, and ten at 50. The
count — the only thing that moved — is the exact quantity GitHub throws away, and the summary
line that does carry it (0 retries / 22 retries) is a plain log line inside a job that passed,
which nobody has a reason to open. The ASB problem was found by opening one anyway, by accident.

What matters

The absolute retry count mattered less than the change in it. The count stepped 1 → 22
between two adjacent main runs and then held at exactly 22 for four runs. That step was the real
signal, and no baseline existed anywhere to notice it against.

Proposal

1. Job summary. Each supervised run appends a retry ledger to $GITHUB_STEP_SUMMARY, so the
   numbers are on the run page without opening a log, and without competing for the ten annotation
   slots.
2. A counted annotation, emitted first. One ::warning carrying the count, before the per-test
   [FLAKY] warnings, so it is inside the ten that survive.
3. Run-over-run. Each job publishes a machine-readable ledger artifact; a roll-up job aggregates
   the run and diffs it against the last completed main run, so 1 → 22 is reported as a delta
   rather than as a number nobody has a baseline for.

Deliberately not proposed: failing the build past a retry threshold. An earlier suggestion of
"fail past 5" is too loose — a suite legitimately at 3 today would be one bad day from a red main,
and the value here is visibility, not a cliff. Any nonzero count should be something a human
explains; that is a review norm, not a gate.

Related

• Wait for the ASB emulator's management api, and pin the image #3783 — the ASB emulator readiness gate that was the actual cause of the 22.
• Restore 32 Azure Service Bus tests, re-tag 11 that are genuinely broken (GH-3763, GH-3786) #3784 — restoring the 32 tests the flaky tags had hidden.
• Flaky-tag removal campaign: the Category=Flaky exclusion list is a debt ledger #3763 — the flaky-tag debt ledger.