Wait for the ASB emulator's management api, and pin the image - #3783
Merged
Conversation
…follow-up)
CIAzureServiceBus has been reporting green while burning 22 of its 25-retry
budget on every single run -- 85% of all retries in the repository -- and all 22
were one class, InlineSendingAndReceivingCompliance, failing identically four
consecutive main runs in a row. Nothing about that is flaky.
With retries forced off in CI, the real failure surfaced:
Azure.Messaging.ServiceBus.ServiceBusException :
Service is warming up. Please try after some time.
Status: 503 (Service Unavailable) SystemTracker: localhost:$Resources/topics
The emulator binds its AMQP listener on 5673 as soon as Kestrel starts, but its
MANAGEMENT api on 5300 -- the one ServiceBusAdministrationClient uses to create
queues and topics -- answers 503 for another ~26 seconds. The readiness gate was
a bare TCP connect to 5673, so it announced "up and ready!" in under a second
while every provisioning call still failed. InlineComplianceFixture threw out of
InitializeAsync, which fails all 22 tests in the class at once, and the retry
policy reran each of them alone in a fresh process -- warm by then -- so the job
went green and nobody saw it.
Nothing in Wolverine caused this. servicebus-emulator:latest moved 2.0.0 ->
2.0.1, and 2.0.1 warms more slowly. CI pulls :latest fresh every run and got the
new image; a developer machine keeps whatever it pulled months ago, which is why
this was never reproducible locally and why it appeared to land in a tidy
three-commit window with no plausible commit in it. Reverting the only candidate
in that window (GH-3774) changed nothing -- verified with a two-arm CI probe.
So, both halves:
- The gate now polls the management endpoint until it stops answering 503, and
THROWS instead of logging a warning and carrying on. The old behaviour meant an
emulator that never started at all still fed the whole suite into a broker that
could not serve it, and the failures read as flaky tests rather than as
infrastructure that never came up. Budget raised 60s -> 3 minutes; the measured
warm-up is ~26s on a fast machine and a 4-vCPU runner is slower.
- Both images pinned. The emulator to 2.0.1, which is what :latest resolves to
today. azure-sql-edge publishes no concrete tag matching its :latest manifest,
so it is pinned by digest -- byte-for-byte what CI has already been running, so
it changes nothing except that it can no longer move underneath us.
Verified locally against the exact CI image (pulled 2.0.1, removed the containers
for a genuine cold start): the gate waits 26s and gets a 200 where it previously
returned instantly on the TCP connect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA
jeremydmiller
added a commit
that referenced
this pull request
Aug 2, 2026
…-3786) The March 2026 sweep turned out to be neither wholly stale nor wholly sound. Measured, on a freshly recreated emulator (2.0.1) with the GH-3783 readiness gate and retries off -- 304 tests, 24 failed, 67.6 minutes: CLEAN, stay untagged 4 classes, 32 tests, 2.2 min BROKEN, tagged again 11 classes, 29 tests, 57.0 min Every one of the 11 is a conventional-routing test, and 21 of the 24 failures carry the identical `BrokerInitializationException : Unable to initialize the Broker asb in time`, each burning ~2m15s on the timeout before failing. That is not flakiness: it is 100% failure, deterministically, and it has been hidden since 2026-03-21. Filed as GH-3786. Each re-tag now carries its measured numbers in a comment beside it. Not one of the original 19 tags recorded a reason, a number or an issue link, which is precisely why nobody could tell a hard failure from a timing race -- and why 21% of this project sat dark for four months. Restored: BufferedSendingAndReceivingCompliance (23 tests, 0.36m), StatefulResourceSmokeTests, Bug_2283_purge_session_subscription, and Bug_1684_separated_handlers_and_conventional_routing. Discovery goes from 239 tests running to 271; hidden drops from 65 to 33. No sharding. The earlier 20-minute cap breach was 57 minutes of broker timeouts, not capacity -- without the broken classes the suite measures 10.7 minutes, which is what the job already runs. Splitting it would have spread the timeouts across three jobs and hidden them a second time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA
This was referenced Aug 2, 2026
This was referenced Aug 4, 2026
erdtsieck
pushed a commit
to erdtsieck/wolverine
that referenced
this pull request
Aug 4, 2026
…JasperFxGH-3763) Two failure modes that have each already cost this repository real time, and both were still live. ## Readiness gates Seven of the eight gates logged a warning and carried on. That is precisely the shape that made CIAzureServiceBus spend 22 of its 25-retry budget on four consecutive GREEN main runs: the gate declared the emulator ready in under a second, provisioning ran against a management api still answering 503, a class fixture threw, and 22 tests failed together -- reported as flaky tests rather than as infrastructure that never came up. It was found by accident. JasperFxGH-3783 made that one gate fatal and left the rest. They now share one path, awaitService, which throws when its budget runs out and names the service, the budget, the attempt count and the last real error. A single path matters as much as the behaviour: a gate added later cannot quietly reintroduce warn-and-continue. Worth singling out, because it was the thinnest by a wide margin: the Postgres gate allowed ten attempts separated by 250ms -- a 2.5 SECOND budget for a container start -- and then logged an error and let the suite run anyway. It was already running out of room in production. In CIMQTT5 on main run 30847233633 it spent four of its ten attempts before Postgres answered, roughly 1.1s of its 2.5s allowance, so a slower runner would have sailed past the end and started tests against a database that was not up. Every failure after that would have looked like a test problem. Two probes changed shape rather than only their ending: - Pub/Sub went from a TCP connect to an HTTP request, so it asks the emulator a question instead of only checking that something is listening. - The attempt-count budgets became wall-clock deadlines. "60 seconds" in the old SQL Server warning was never true: each failed attempt could burn the connection's own 5s timeout before the 2s sleep, making 30 attempts anywhere from 60s to 210s. Kafka is knowingly still TCP-only -- a broker accepts connections before it will serve metadata -- because a real probe needs Confluent.Kafka, which this build project does not reference. Filed separately; making it fatal is the part that matters today. The ASB gate moved onto the shared path with its semantics preserved exactly, including keying on "not 503" rather than "== 200": CI answers 400 there and a developer machine answers 200, so asserting 200 would hang for the whole budget and then fail the job. ## Image pins Every remaining rolling tag that CI pulls is now pinned, each to exactly what the rolling tag resolved to at the time, so nothing about what CI runs changes -- it just stops changing on its own. - apachepulsar/pulsar :latest -> 4.2.4, in two places. This one was live: :latest moved to 4.2.4 on 2026-08-03, the same day. docker-compose.yml said 4.0.3, so CI (which uses Testcontainers) and a developer running docker compose were on different Pulsar versions and only one of them could move without warning. Both now say 4.2.4. - nats :latest -> 2.14.4, in three places, via one shared constant. - fsouza/fake-gcs-server :latest -> 1.55.1 (same digest). - mcr.microsoft.com/mssql/server:2025-latest -> pinned by DIGEST, because no concrete 2025-CU* tag matches that manifest -- the same reason azure-sql-edge is pinned by digest. Not changed: jaegertracing/all-in-one:latest in src/Testing/OpenTelemetry/docker-compose.yml. That file is referenced by no build target and no workflow -- it is a local tracing demo, so it is not a CI exposure. Verified locally: the shared throw path reports "BogusService was not ready after 3s (3 attempts). Last attempt: Connection refused", and the rewritten Pub/Sub and Postgres gates both pass against live containers. The remaining gates are exercised by CI itself, since every job starts its services through them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
This was referenced Aug 4, 2026
This was referenced Aug 5, 2026
portlogicsvn
pushed a commit
to portlogicsvn/wolverine
that referenced
this pull request
Aug 6, 2026
…perFxGH-3814) Split out of JasperFxGH-3813, which made every readiness gate fatal but left this one's probe unchanged. WaitForKafkaToBeReady opened a TCP connection to localhost:9092 and called that ready. Measured against a cold broker three times, that gate is not merely weak, it is effectively a no-op: TCP connect succeeded at 0.0s <-- old gate passed HERE Metadata served at 3.0s <-- broker actually usable HERE The TCP connect succeeds at 0.0s every time because Docker's port proxy accepts as soon as the container is created, before the broker is listening inside it. Metadata was not served until 2.4-3.0s across the three runs. So the gate passed instantly and unconditionally, then handed a ~3 second window to a suite that immediately creates topics -- and "Leader not available", "Not enough replicas" and topic-creation errors all read as flaky tests rather than as a broker that was not up. Same defect class as the ASB emulator gate in JasperFxGH-3783, which cost 22 retries per run across four green main runs before it was noticed by accident. Probes with AdminClient.GetMetadata instead, which round-trips the Kafka protocol and does not answer until the broker will serve. Also treats an empty broker list as not-ready, since a broker mid-startup can answer metadata with an empty cluster view and still refuse a topic creation. Confluent.Kafka is added to the build project and centrally versioned at 2.14.0, matching the Confluent.SchemaRegistry.Serdes.* packages already in the tree. That dependency addition is the reason this was not folded into JasperFxGH-3813. librdkafka's stderr chatter is silenced because awaitService already reports every attempt and carries the last real failure into its final error. Verified by running the CIKafka target against a deliberately removed Kafka container: the gate waited and passed rather than timing out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up from the #3781 work: while measuring CI to plan the flakiness burn-down,
CIAzureServiceBusturned out to be reporting green while burning 22 of its 25-retry budget every run — 85% of all retries in the repository, all of them one class.It was never flaky
0bf49b4f83ad7986b61dfdd42a146dd558c2141d948800d95ad0bf22/22 identical, four runs running, always
InlineSendingAndReceivingCompliance.The actual failure
Forcing retries off in CI surfaced it:
The emulator binds AMQP on 5673 as soon as Kestrel starts, but the management api on 5300 — the one
ServiceBusAdministrationClientuses to provision — answers 503 for another ~26 seconds.WaitForAzureServiceBusEmulatorToBeReadywas a bareTcpClient.Connect("localhost", 5673), so it printed "up and ready!" in under a second while provisioning still failed.InlineComplianceFixturethrew out ofInitializeAsync→ xUnit fails all 22 tests in the class → Bobcat reran each alone in a fresh process → warm by then → green.No commit caused it
servicebus-emulator:latestmoved 2.0.0 → 2.0.1, and 2.0.1 warms more slowly::latesttodaysha256:5a96d8…= 2.0.1sha256:a00c96…= 2.0.0CI pulls fresh every run; developer machines don't. That explains the deterministic CI failure, the total non-reproduction locally, and the tidy-looking three-commit window with no plausible commit in it. I ran a two-arm CI probe reverting the only candidate in that window (GH-3774) — no effect, exactly as this explanation predicts.
Changes
2.0.1(what:latestis today).azure-sql-edge→ by digest, since no concrete tag matches its:latestmanifest; byte-for-byte what CI already runs, so nothing changes except that it can't move.Verification
Pulled 2.0.1 locally and removed the containers for a genuine cold start — the first time this machine could reproduce CI's conditions. The gate waits 26s and gets a 200 where it previously returned instantly.
Expected on this PR:
CIAzureServiceBusreports 0 retries instead of 22, and drops ~3 minutes (22 fresh-process retries at ~8-9s each) off what is currently the longest job in the matrix.Not in this PR
WaitForPubsubEmulatorToBeReadyand the Kafka gate are the same TCP-only, warn-and-continue shape, and:latestis used more widely in the compose file. Neither is currently costing retries, so they're a follow-up sweep rather than scope creep here.🤖 Generated with Claude Code
https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA