CIRabbitMQ has become a wall-clock pole: 749s -> 1023s in a day, with high run-to-run variance

[jeremydmiller]

Observation with a caveat, filed so it is written down somewhere rather than noticed twice.

What the numbers show

CIRabbitMQ duration across the green main runs of 2026-08-03, oldest first:

run	CIRabbitMQ	CIAzureServiceBus
21:59	749s	1032s
21:59	770s	1005s
21:59	910s	1010s
21:59	912s	1007s
23:37	911s	1008s
01:33	1023s	1012s

Against the 20-minute (1200s) job cap, CIRabbitMQ has gone from 62% to 85% of the cap and is now level with CIAzureServiceBus as the wall-clock pole.

The caveat, stated plainly

Four of those runs were concurrent (a burst of merges), so GitHub runner contention is in play — and CIRabbitMQ's spread within that single concurrent batch was 749–912s, a 22% swing. So 1023s may be inside its own noise band.

What makes it worth recording anyway: CIAzureServiceBus over the same window was 1005–1032s, essentially flat. Whatever moved CIRabbitMQ did not move ASB, which is not what pure runner contention looks like.

Why it matters

• It is now one of the two jobs setting tests.yml wall clock.
• It is also the job carrying the newly-visible flakes — Bug_189_fails_if_there_are_many_messages_in_queue_on_startup and ConventionalRouting.end_to_end_with_conventional_routing, both TimeoutException. A suite getting slower and a suite timing out more are plausibly the same story.
• A job at 85% of a hard cap has little room; the ASB experience (Shard CIAzureServiceBus: the reason sharding was abandoned expired when GH-3786 removed the broker timeouts #3790) is that the cap is reached without much warning.

Suggested next step

Do not act on this yet — measure it on a quiet main run first, uncontended, and compare per-class durations via a TRX logger rather than job totals. If the increase is real, the two TimeoutException flakes are the first place to look.

Related: #3752 (wall-clock poles), #3790 (sharding ASB).