Make every readiness gate fatal, and pin the images that still rolled (GH-3763) - #3813
Merged
Merged
Conversation
…GH-3763) Two failure modes that have each already cost this repository real time, and both were still live. ## Readiness gates Seven of the eight gates logged a warning and carried on. That is precisely the shape that made CIAzureServiceBus spend 22 of its 25-retry budget on four consecutive GREEN main runs: the gate declared the emulator ready in under a second, provisioning ran against a management api still answering 503, a class fixture threw, and 22 tests failed together -- reported as flaky tests rather than as infrastructure that never came up. It was found by accident. GH-3783 made that one gate fatal and left the rest. They now share one path, awaitService, which throws when its budget runs out and names the service, the budget, the attempt count and the last real error. A single path matters as much as the behaviour: a gate added later cannot quietly reintroduce warn-and-continue. Worth singling out, because it was the thinnest by a wide margin: the Postgres gate allowed ten attempts separated by 250ms -- a 2.5 SECOND budget for a container start -- and then logged an error and let the suite run anyway. It was already running out of room in production. In CIMQTT5 on main run 30847233633 it spent four of its ten attempts before Postgres answered, roughly 1.1s of its 2.5s allowance, so a slower runner would have sailed past the end and started tests against a database that was not up. Every failure after that would have looked like a test problem. Two probes changed shape rather than only their ending: - Pub/Sub went from a TCP connect to an HTTP request, so it asks the emulator a question instead of only checking that something is listening. - The attempt-count budgets became wall-clock deadlines. "60 seconds" in the old SQL Server warning was never true: each failed attempt could burn the connection's own 5s timeout before the 2s sleep, making 30 attempts anywhere from 60s to 210s. Kafka is knowingly still TCP-only -- a broker accepts connections before it will serve metadata -- because a real probe needs Confluent.Kafka, which this build project does not reference. Filed separately; making it fatal is the part that matters today. The ASB gate moved onto the shared path with its semantics preserved exactly, including keying on "not 503" rather than "== 200": CI answers 400 there and a developer machine answers 200, so asserting 200 would hang for the whole budget and then fail the job. ## Image pins Every remaining rolling tag that CI pulls is now pinned, each to exactly what the rolling tag resolved to at the time, so nothing about what CI runs changes -- it just stops changing on its own. - apachepulsar/pulsar :latest -> 4.2.4, in two places. This one was live: :latest moved to 4.2.4 on 2026-08-03, the same day. docker-compose.yml said 4.0.3, so CI (which uses Testcontainers) and a developer running docker compose were on different Pulsar versions and only one of them could move without warning. Both now say 4.2.4. - nats :latest -> 2.14.4, in three places, via one shared constant. - fsouza/fake-gcs-server :latest -> 1.55.1 (same digest). - mcr.microsoft.com/mssql/server:2025-latest -> pinned by DIGEST, because no concrete 2025-CU* tag matches that manifest -- the same reason azure-sql-edge is pinned by digest. Not changed: jaegertracing/all-in-one:latest in src/Testing/OpenTelemetry/docker-compose.yml. That file is referenced by no build target and no workflow -- it is a local tracing demo, so it is not a CI exposure. Verified locally: the shared throw path reports "BogusService was not ready after 3s (3 attempts). Last attempt: Connection refused", and the rewritten Pub/Sub and Postgres gates both pass against live containers. The remaining gates are exercised by CI itself, since every job starts its services through them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
…ling
The digest pinned in the previous commit does not exist as a pullable manifest,
and every job that starts SQL Server failed on it:
sqlserver Error manifest for mcr.microsoft.com/mssql/server@sha256:aff82a72...
not found: manifest unknown
Nine jobs, one cause -- CISqlServer, CIPersistence, CIEfCore, CIPolecat,
CIPolecatSagas, CIPolecatWorkflow, CIMQTT, CIMQTT5 and CIRabbitMQ, which starts
sqlserver alongside rabbitmq and postgresql.
The digest came from a curl HEAD of the registry's manifest endpoint with an
Accept for a manifest LIST. mssql/server:2025-latest is a single-platform
manifest, so MCR answered with a digest for a representation that is not
pullable. `docker buildx imagetools inspect` reports the real one:
sha256:86cc6144ef39bb0fbed2329e1ad79b13ee82e7b2e4739213a0db0800e668a74a
The rule, now written next to the pin: resolve a digest by asking the tool that
will do the pulling, never by hand-rolling a registry request.
Verified this time rather than assumed -- `docker manifest inspect` by digest
resolves, `docker compose pull sqlserver` completes, and every other image in
the compose file was re-checked the same way, including the ones this branch
did not touch.
Also aligns docker-compose's nats to 2.14.4, matching
NatsContainerFixture.NatsImage. "nats:2" is a floating minor tag that happens to
equal 2.14.4 today; same one-version-per-service reasoning as Pulsar.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two failure modes that have each already cost this repo real time, both still live.
1. Readiness gates
Seven of eight gates logged a warning and carried on. That is precisely the shape that made
CIAzureServiceBusspend 22 of its 25-retry budget on four consecutive green main runs — the gate declared the emulator ready in under a second, provisioning ran against a management API still answering 503, a class fixture threw, and 22 tests failed together, reported as flaky tests rather than infrastructure that never came up. It was found by accident. #3783 made that one gate fatal and left the rest.They now share one path,
awaitService, which throws when its budget expires and names the service, budget, attempt count and last real error. The single path matters as much as the behaviour — a gate added later cannot quietly reintroduce warn-and-continue.The Postgres gate was already failing in production
Ten attempts separated by 250ms is a 2.5 second budget for a container start, after which it logged an error and let the suite run anyway. In CIMQTT5 on main run 30847233633 it spent four of its ten attempts before Postgres answered — ~1.1s of its 2.5s allowance:
A slower runner sails past the end and starts tests against a database that is not up. Every failure after that looks like a test problem.
Two probes changed shape, not just their ending
"60 seconds"was never true — each failed attempt could burn the connection's own 5s timeout before the 2s sleep, making 30 attempts anywhere from 60s to 210s.Kafka is knowingly still TCP-only — a broker accepts connections before it will serve metadata — because a real probe needs
Confluent.Kafka, which this build project does not reference. Filed separately; making it fatal is what matters today.The ASB gate keeps its semantics exactly, including keying on not 503 rather than == 200: CI answers 400 there and a dev machine answers 200, so asserting 200 would hang the full budget and fail the job.
2. Image pins
Every remaining rolling tag CI pulls is pinned to exactly what it resolved to, so nothing about what CI runs changes — it just stops changing on its own.
apachepulsar/pulsar(×2):latest4.2.4nats(×3):latest2.14.4(one shared constant)fsouza/fake-gcs-server:latest1.55.1(same digest)mcr.microsoft.com/mssql/server2025-latestPulsar was live.
:latestmoved to 4.2.4 on 2026-08-03 — the same day — whiledocker-compose.ymlsaid4.0.3. CI uses Testcontainers, so CI and a developer runningdocker compose upwere on different Pulsar versions, and only one of them could move without warning. Both now say 4.2.4. That is the concrete half of #3799.SQL Server is pinned by digest because no concrete
2025-CU*tag matches the2025-latestmanifest — the same reasonazure-sql-edgeis pinned by digest.Not changed:
jaegertracing/all-in-one:latestinsrc/Testing/OpenTelemetry/docker-compose.yml. That file is referenced by no build target and no workflow — a local tracing demo, not a CI exposure. Calling it out so "no rolling tags remain" is not overclaimed.Verification
Throw path, the reshaped Pub/Sub probe, and the Postgres gate (which lives in
build.csand proves the cross-file wiring) all verified against live containers. The remaining gates are exercised by CI itself — every job starts its services through them, so a wrong probe fails that specific job in this PR rather than hiding.🤖 Generated with Claude Code
https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m