Skip to content

Retry docker compose up so a registry timeout does not redden main - #3807

Merged
jeremydmiller merged 1 commit into
mainfrom
build/docker-compose-pull-retry
Aug 3, 2026
Merged

Retry docker compose up so a registry timeout does not redden main#3807
jeremydmiller merged 1 commit into
mainfrom
build/docker-compose-pull-retry

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Three of the four red main runs on 2026-08-03 were not test failures. CIOtel (twice, runs 30835955313 and 30782661294) and CIMQTT5 (30843486384) each died in under a minute, before a single test ran:

rabbitmq Error Get "https://registry-1.docker.io/v2/": context deadline exceeded

docker compose up contacts the registry for any image that is not already cached, and GitHub's runners time out against Docker Hub often enough to do this on their own. LaunchDockerServices fired exactly one attempt and asserted a zero exit code, so a single blip failed the whole job.

The change

ComposeUp makes three attempts with a 5s/10s backoff, logging the tool's own output on each failure, and throws the real message when they are exhausted.

Every failure is retried, not just registry-shaped ones. Matching on message text would be guessing at a moving target. A genuinely broken compose file still fails — it just takes three tries to get there, and the exception carries the real reason either way. Each failed attempt logs the output, so the job log states which case it was rather than leaving the reader to infer it.

DockerUp in build.cs was a byte-for-byte duplicate of the same single-shot logic and pulls every image in the compose file, making it the most exposed caller of all, so it now shares the helper.

Verification

Run against a nonexistent service via a temporary target — two warnings, then the genuine error, and a total elapsed time of 0:15 confirming the 5s + 10s backoff actually happened:

[WRN] docker compose up failed for no-such-service (attempt 1 of 3), retrying in 5s:
      no such service: no-such-service
[WRN] docker compose up failed for no-such-service (attempt 2 of 3), retrying in 10s:
      no such service: no-such-service
[ERR] docker compose up failed for no-such-service after 3 attempts:
      no such service: no-such-service
ProcessException: Process 'docker' exited with code 1.
   Error output:
      no such service: no-such-service

ProbeRetry         Failed          0:15

The success path is unchanged apart from a log line when it took more than one attempt.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m

Three of the four red main runs on 2026-08-03 were not test failures. CIOtel
(twice) and CIMQTT5 died in under a minute on

    Get "https://registry-1.docker.io/v2/": context deadline exceeded

before a single test ran. `docker compose up` reaches out to the registry for
any image that is not already cached, and GitHub's runners time out against
Docker Hub often enough to do this on their own. LaunchDockerServices fired one
attempt and asserted a zero exit code, so one blip failed the whole job.

A pull timeout is transient and says nothing about the commit that triggered
it, so it should cost a retry rather than a red build. ComposeUp now makes
three attempts with a 5s/10s backoff and logs the tool's own output on each
failure.

Every failure is retried, not just registry-shaped ones: matching on the
message text would be guessing at a moving target, and a genuinely broken
compose file still fails, just after three tries, with the real message
carried by the exception either way.

DockerUp in build.cs was a byte-for-byte duplicate of the same single-shot
logic, and pulls every image in the compose file, so it routes through the
same helper.

Verified with a temporary target against a nonexistent service: two warnings,
then the genuine "no such service" error, total elapsed 0:15 confirming the
backoff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant