Restore 32 Azure Service Bus tests, re-tag 11 that are genuinely broken (GH-3763, GH-3786) - #3784
Merged
Merged
Conversation
…H-3763) 19 [Trait("Category", "Flaky")] tags hid 65 of the 304 tests in this project -- 21% of the suite never ran. Not one of them recorded a reason, a number, or an issue link, so there was no way to tell from the code whether any had ever been real. git log -L on each tag line says most were never individually judged: 15 of the 19 went in on 2026-03-20/21 across four commits with messages like "Tag all ConventionalRouting tests as Flaky" and "tag flaky AWS/Azure SB tests". That is a bulk sweep, not 15 investigations. The other 4 were added later alongside real PRs (#2588, #3103, #3534) and are left tagged here to be judged one at a time. There is now a specific reason to think the sweep was measuring infrastructure rather than tests. GH-3781 follow-up found the emulator readiness gate was TCP-connecting to the AMQP port while provisioning goes to the management port, which answers 503 for ~26 seconds after the socket opens -- so any class doing provisioning near the start of a run could fail for reasons that had nothing to do with it. Nine of the fifteen are ConventionalRouting discovery tests, which are exactly that shape. This commit only removes the tags. Whether they stay off is decided by what CI says, per class, by name -- the RabbitMQ pass in #3780 nearly untagged a class that fails 2 of 20 every run because it was verified as part of a set rather than individually. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA
…-3786) The March 2026 sweep turned out to be neither wholly stale nor wholly sound. Measured, on a freshly recreated emulator (2.0.1) with the GH-3783 readiness gate and retries off -- 304 tests, 24 failed, 67.6 minutes: CLEAN, stay untagged 4 classes, 32 tests, 2.2 min BROKEN, tagged again 11 classes, 29 tests, 57.0 min Every one of the 11 is a conventional-routing test, and 21 of the 24 failures carry the identical `BrokerInitializationException : Unable to initialize the Broker asb in time`, each burning ~2m15s on the timeout before failing. That is not flakiness: it is 100% failure, deterministically, and it has been hidden since 2026-03-21. Filed as GH-3786. Each re-tag now carries its measured numbers in a comment beside it. Not one of the original 19 tags recorded a reason, a number or an issue link, which is precisely why nobody could tell a hard failure from a timing race -- and why 21% of this project sat dark for four months. Restored: BufferedSendingAndReceivingCompliance (23 tests, 0.36m), StatefulResourceSmokeTests, Bug_2283_purge_session_subscription, and Bug_1684_separated_handlers_and_conventional_routing. Discovery goes from 239 tests running to 271; hidden drops from 65 to 33. No sharding. The earlier 20-minute cap breach was 57 minutes of broker timeouts, not capacity -- without the broken classes the suite measures 10.7 minutes, which is what the job already runs. Splitting it would have spread the timeouts across three jobs and hidden them a second time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA
jeremydmiller
changed the title
jeremydmiller
marked this pull request as ready for review
August 2, 2026 21:01
This was referenced Aug 2, 2026
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
19
[Trait("Category", "Flaky")]tags hid 65 of 304 tests in this project — 21% of the suite never ran. None of them recorded a reason, a number, or an issue link, so there was no way to tell whether any had ever been real.git log -Lon each tag line showed 15 of the 19 went in on 2026-03-20/21 across four commits with messages like "Tag all ConventionalRouting tests as Flaky" — a bulk sweep, not 15 investigations. This PR untagged all 15 and measured what actually happens.The measurement
Full run,
net9.0Release, serial, freshly recreated emulator (servicebus-emulator:2.0.1) with the #3783 readiness gate, retries off so nothing is retried away: 304 tests, 24 failed, 67.6 minutes.Restored:
BufferedSendingAndReceivingCompliance(23 tests, 0.36m),StatefulResourceSmokeTests,Bug_2283_purge_session_subscription,Bug_1684_separated_handlers_and_conventional_routing.Re-tagged, each now carrying its measured numbers in a comment beside the tag: all nine
ConventionalRoutingclasses,Bug_1933_multi_tenant_conventional_routing,Bug_2307_batching_with_conventional_routing, andend_to_end(2 of 6).Discovery: 239 → 271 tests running, hidden 65 → 33.
These are not flaky, they are broken — #3786
Every one of the 11 is a conventional-routing test, and 21 of the 24 failures carry the identical
BrokerInitializationException : Unable to initialize the Broker asb in time, each burning ~2m15s on the timeout. 100% failure, deterministic, on a clean emulator with the readiness fix already in.Filed as #3786. A useful starting clue is in there:
ConventionalRouting.when_using_handler_type_namingwas never tagged, has been running in CI all along, and passes — whatever it does differently from its eleven neighbours is probably the shortest path to the cause.No sharding
The first run of this branch breached the 20-minute cap and I was about to shard the job. The measurement says don't: 57 of the 68 minutes are broker timeouts, not capacity. Without the broken classes the suite measures 10.7 minutes, which is what
CIAzureServiceBusalready runs. Sharding would have spread the timeouts across three jobs and hidden them a second time.Why the tags now carry numbers
The whole reason this took a measurement rather than a judgement is that the original tags were bare. Every surviving tag now says what fails, how often, how long it takes, and which issue tracks it — so the next person can re-judge it in seconds instead of four months.