Retry docker compose up so a registry timeout does not redden main - #3807
Merged
Conversation
Three of the four red main runs on 2026-08-03 were not test failures. CIOtel
(twice) and CIMQTT5 died in under a minute on
Get "https://registry-1.docker.io/v2/": context deadline exceeded
before a single test ran. `docker compose up` reaches out to the registry for
any image that is not already cached, and GitHub's runners time out against
Docker Hub often enough to do this on their own. LaunchDockerServices fired one
attempt and asserted a zero exit code, so one blip failed the whole job.
A pull timeout is transient and says nothing about the commit that triggered
it, so it should cost a retry rather than a red build. ComposeUp now makes
three attempts with a 5s/10s backoff and logs the tool's own output on each
failure.
Every failure is retried, not just registry-shaped ones: matching on the
message text would be guessing at a moving target, and a genuinely broken
compose file still fails, just after three tries, with the real message
carried by the exception either way.
DockerUp in build.cs was a byte-for-byte duplicate of the same single-shot
logic, and pulls every image in the compose file, so it routes through the
same helper.
Verified with a temporary target against a nonexistent service: two warnings,
then the genuine "no such service" error, total elapsed 0:15 confirming the
backoff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three of the four red
mainruns on 2026-08-03 were not test failures. CIOtel (twice, runs 30835955313 and 30782661294) and CIMQTT5 (30843486384) each died in under a minute, before a single test ran:docker compose upcontacts the registry for any image that is not already cached, and GitHub's runners time out against Docker Hub often enough to do this on their own.LaunchDockerServicesfired exactly one attempt and asserted a zero exit code, so a single blip failed the whole job.The change
ComposeUpmakes three attempts with a 5s/10s backoff, logging the tool's own output on each failure, and throws the real message when they are exhausted.Every failure is retried, not just registry-shaped ones. Matching on message text would be guessing at a moving target. A genuinely broken compose file still fails — it just takes three tries to get there, and the exception carries the real reason either way. Each failed attempt logs the output, so the job log states which case it was rather than leaving the reader to infer it.
DockerUpinbuild.cswas a byte-for-byte duplicate of the same single-shot logic and pulls every image in the compose file, making it the most exposed caller of all, so it now shares the helper.Verification
Run against a nonexistent service via a temporary target — two warnings, then the genuine error, and a total elapsed time of
0:15confirming the 5s + 10s backoff actually happened:The success path is unchanged apart from a log line when it took more than one attempt.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m