ci: pre-pull and cache heavy e2e images via GitHub Actions cache by glours · Pull Request #14080 · docker/compose

glours · 2026-08-17T14:45:54Z

What I did

Added a GHA cache layer for the passive Docker images used by the e2e suite.

A new allowlist .github/e2e-images.txt lists the infrastructure images that every test cell downloads: docker:dind, mariadb, golang:alpine, fluent/fluent-bit:3.1.7-debug, registry:3, and the gtardif/sentences-* / nginx:alpine helper images.

Before each test run the workflow restores a pre-built tar from the GHA cache (key = Docker engine version + hash of the image list). On a cache hit the images are loaded instantly with docker load; on a miss they are pulled in parallel (xargs -P 4 -n 1 docker pull) and saved for the next run. Any change to .github/e2e-images.txt automatically invalidates the cache.

Images used by pull-policy and image-identity assertions (alpine:* and several fixture-specific tags) are intentionally absent from the list so those tests remain unaffected. No change to local make e2e-compose or make e2e-compose-standalone runs.

Note: this PR is stacked on #14073. Until that PR merges into main the diff includes both sets of changes.

The ci.yml changes cannot be pushed directly from the sandbox environment (requires workflow OAuth scope). They are provided as .github/ci-workflow-cache-only.patch. To apply:
git apply .github/ci-workflow-cache-only.patch
git rm .github/ci-workflow-cache-only.patch
git commit --amend --no-edit --signoff
git push --force-with-lease

Related issue

No related issue — this is a CI infrastructure improvement.

(not mandatory) A picture of a cute animal, if possible in relation to what you did

A panda that found its bamboo already pre-stocked instead of having to go fetch it 🐼

docker-agent

🟢 No issues found — LGTM! View logs.

…igurable Several e2e tests used fixed host ports (8070, 8080, 8090, etc.), global buildx builders registered with --use, and a hardcoded IPC container name. Under parallel execution these shared daemon-level resources caused port conflicts, buildkitd container-name collisions, and intermittent failures. Separately, the standalone make target hardcoded -parallel=1, forcing the entire standalone suite to run serially even though the same tests run in parallel in plugin mode. Changes: - Convert fixed host ports in four fixtures (network-test, sentences, build-test, volume-test) to ephemeral bindings; add ServicePublishedPort helper to framework.go to resolve the actual mapped port at runtime. - Give each buildx-builder test a unique daemon-scoped name via BuilderName helper, preventing container-name collisions between parallel goroutines. Applies to TestBuildPlatformsWithCorrectBuildxConfig, TestBuildPrivileged, TestBuildBuilder, TestBuildEntitlements, TestBuildTLS. - Rename the fixed ipc_mode_container to a project-scoped name (ipc_e2e-src) and update the ipc-test fixture accordingly. - Replace the hardcoded -parallel=1 in make e2e-compose-standalone with E2E_PARALLEL_PLUGIN?=4 and E2E_STANDALONE_PARALLEL?=4, aligning both modes with the effective GOMAXPROCS on 4-core GitHub Actions runners. Both variables are overridable (E2E_STANDALONE_PARALLEL=1 restores serial local debugging). - Raise the poll delay in watch_test.go long-running polls (90s/120s timeout) from the default 100ms to 1s; bump cascade_test.go and publish_test.go from 100ms to 500ms. Reduces CPU pressure without affecting effective test duration. The logging-driver fixture retains its fixed port 24224: the Docker fluentd log driver encodes fluentd-address at service-start time, making ephemeral resolution non-trivial. TestLoggingDriver is serial so there is no collision risk in practice. Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>

docker-agent

Assessment: 🟡 NEEDS ATTENTION

Two findings in the newly added code — one in pkg/e2e/framework.go and one in the Makefile — affect correctness of local developer runs and the new test helper.

docker-agent · 2026-08-18T08:12:08Z

 TEST_FLAGS?=
 E2E_TEST?=
+E2E_PARALLEL_PLUGIN?=4
+E2E_STANDALONE_PARALLEL?=4


[medium] E2E_STANDALONE_PARALLEL defaults to 4, silently removing the serial guarantee for local make e2e-compose-standalone runs

The previous command hardcoded -parallel=1; standalone tests have always been run serially. This PR changes the default to 4, meaning any developer who runs make e2e-compose-standalone without explicitly setting E2E_STANDALONE_PARALLEL=1 will now get 4 parallel workers.

The PR description says "make e2e-compose-standalone unchanged for local runs", but this is a behaviour change. Standalone tests that rely on unique ports, container names, or Docker-daemon-global state (e.g., build caches, named builders) can fail intermittently when run 4-way in parallel. The recent history of this file also shows the standalone runner was previously kept serial for stability.

If the intent is to let CI run at higher parallelism while keeping local runs safe, the default should stay at 1 and only CI should override via the environment:

Suggested change

E2E_STANDALONE_PARALLEL?=4

E2E_STANDALONE_PARALLEL?=1

Confidence Score

🟡 moderate 67/100

codecov · 2026-08-18T08:31:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Add a GHA cache layer for the passive Docker images used by the e2e suite. Before each test run the workflow attempts to restore a pre-built tar from the cache (key = Docker engine version + sha256 of the image list). On a cache hit the images are loaded with 'docker load'; on a miss they are pulled in parallel ('xargs -P 4 -n 1') and saved for the next run. A new allowlist .github/e2e-images.txt controls which images are cached. It includes the infrastructure images that every test cell downloads: docker:dind, mariadb, golang:alpine, fluent/fluent-bit:3.1.7-debug, registry:3, and the sentences/nginx helper images. Images used by pull-policy and image-identity assertions (alpine:* and several fixture- specific tags) are intentionally absent so those tests remain unaffected. The ci.yml changes are provided as .github/ci-workflow-cache-only.patch because the sandbox environment lacks the GitHub 'workflow' OAuth scope. Apply with: git apply .github/ci-workflow-cache-only.patch git rm .github/ci-workflow-cache-only.patch git commit --amend --no-edit --signoff Also fix ServicePublishedPort helper to take only the first line of 'compose port' output (SplitN), making it robust when a service is scaled. Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>

The previous patch was generated with Python difflib which omits the required leading space on blank context lines, producing a 'corrupt patch' error when applied with git apply. Regenerated with git diff --cached to get proper unified diff format. Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>

glours · 2026-08-18T10:12:13Z

The measurements didn’t show any meaningful benefit from introducing the cache, so I’m closing the PR.

glours force-pushed the e2e-ci-sharding-and-image-cache branch from d3335a2 to f878666 Compare August 17, 2026 14:47

docker-agent reviewed Aug 17, 2026

View reviewed changes

glours force-pushed the e2e-ci-sharding-and-image-cache branch from f878666 to c0b8477 Compare August 18, 2026 07:43

glours force-pushed the e2e-ci-sharding-and-image-cache branch from c0b8477 to 8e52599 Compare August 18, 2026 07:44

glours requested a review from docker-agent August 18, 2026 08:04

docker-agent reviewed Aug 18, 2026

View reviewed changes

glours force-pushed the e2e-ci-sharding-and-image-cache branch from 9045bd4 to 0f2a67f Compare August 18, 2026 08:58

glours changed the title ~~ci: pre-pull heavy e2e images via GHA cache and shard the test matrix~~ ci: pre-pull and cache heavy e2e images via GitHub Actions cache Aug 18, 2026

glours force-pushed the e2e-ci-sharding-and-image-cache branch from 077f716 to 6602303 Compare August 18, 2026 09:04

glours closed this Aug 18, 2026

Navigation Menu

Uh oh!

ci: pre-pull and cache heavy e2e images via GitHub Actions cache - #14080

ci: pre-pull and cache heavy e2e images via GitHub Actions cache#14080
glours wants to merge 3 commits into
docker:mainfrom
glours:e2e-ci-sharding-and-image-cache

glours commented Aug 17, 2026 •

edited

Loading

Uh oh!

docker-agent left a comment

Uh oh!

docker-agent left a comment

Uh oh!

Uh oh!

docker-agent Aug 18, 2026

Uh oh!

codecov Bot commented Aug 18, 2026

Uh oh!

glours commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

glours commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

docker-agent left a comment

Choose a reason for hiding this comment

Uh oh!

docker-agent left a comment

Choose a reason for hiding this comment

Assessment: 🟡 NEEDS ATTENTION

Uh oh!

Uh oh!

docker-agent Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Aug 18, 2026

Codecov Report

Uh oh!

glours commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

glours commented Aug 17, 2026 •

edited

Loading