[CI] rebuild a PR's docker image only when its build inputs change - #2675
Conversation
Rebuilds were driven by whether the PR diff touched docker paths, so every rerun of a Dockerfile-touching PR paid for an identical multi-arch build. Hash the build inputs, stamp the hash on the published tag as a label, and compare against it: a PR keeps one pr-<num> tag and rebuilds it only when the inputs actually move, when the tag is missing, or on the one-shot rebuild-ci-image label. resolve-ci-image now selects the PR image whenever that tag is current rather than only when this run built it, so reusing an image does not silently fall back to dev.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
yueming-yuan
changed the title
Also scope pull-requests: write to the docker-build call instead of the whole workflow; only that job removes the one-shot label.
| env: | ||
| BASE_SHA: ${{ github.event.pull_request.base.sha || '' }} | ||
| PR_NUMBER: ${{ github.event.pull_request.number || '' }} | ||
| FORCE_REBUILD: ${{ contains(github.event.pull_request.labels.*.name, 'rebuild-ci-image') }} |
There was a problem hiding this comment.
[P2] Read the live force-rebuild label instead of the triggering payload
github.event.pull_request.labels is frozen to the webhook payload for this workflow run. After a successful labeled run removes rebuild-ci-image, using Re-run all jobs creates another attempt of that same run, so this expression still evaluates true and rebuilds the full multi-arch image again. That breaks both the documented one-shot behavior and the rerun-reuse goal. Please query the current live labels for the PR at decision time. Gating only on github.run_attempt == 1 is not sufficient because a first attempt that failed before consuming the label must still rebuild on retry.
| # Reading the inputs label off the published tag needs registry auth. Forks get no | ||
| # secrets and cannot push anyway, so they skip straight to the released image. | ||
| - name: Login to Docker Hub | ||
| if: env.SAME_REPO == 'true' |
There was a problem hiding this comment.
[P2] Defer Docker Hub login until an image is actually needed
This login now runs for every same-repository PR before the CURRENT == BASE early exit. A source-only PR therefore depends on Docker Hub credentials and registry availability even though this job should select dev; an authentication or registry outage can fail the GPU pipeline before the unchanged-input fast path runs. Please compute CURRENT and BASE first, then authenticate only when the workflow needs to inspect the existing PR image tag.
Read the force-rebuild label from live PR state and carry that decision through label consumption, so rerunning a completed labeled workflow reuses the published tag. Compare image inputs before registry authentication, keeping source-only PRs independent of Docker Hub.
3 participants
Summary
Reuse a PR-scoped CI image until its Docker build inputs change.
Motivation
A Dockerfile-touching PR previously rebuilt the same multi-arch image on every rerun or source-only push because CI keyed the build on changed paths. That repeated an expensive build even when the image inputs were unchanged.
Usage
Push Docker-input changes normally; CI publishes or reuses
radixark/miles:pr-<num>. After the repository'srebuild-ci-imagelabel is provisioned, apply it to rebuild unchanged inputs after a moved base image, floating dependency, or corrupt push; a successful forced build removes the label.Design Notes
docker/image_inputs.pyhashes the files consumed by the cu13 build,docker/build.pystamps that hash on the PR tag, anddocker-decidecompares the current hash with the published label. A current tag setstag_available, so every GPU suite keeps using the PR image even when the present run skips its build.rebuild-ci-imagefrom live PR labels, carries that captured decision through the build, and consumes it only after a successful forced build. A completed labeled run therefore reuses its tag when rerun, while a failed build leaves the label available for retry.devwithout touching the registry; same-repository Docker PRs authenticate only to inspect or publish their PR tag, and fork PRs cannot publish.Dockerfile.rocmremains excluded because this workflow builds cu13; moved base images and other floating remote state remain explicit manual-rebuild cases.Verification
pytest -q tests/ci/test/test_run_suite.py tests/fast/test_docker_image_inputs.py: 98 tests passed against prospective tip34f7e6f1, covering workflow seams, input hashing, and image-label parsing.pre-commit run --files .github/workflows/_build-pr-ci-image.yml tests/ci/test/test_run_suite.py: every applicable repository hook passed.go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 .github/workflows/_build-pr-ci-image.yml .github/workflows/pr-test.yml: exited 0 with no diagnostics.Review Focus
docker/image_inputs.py: verify the declared input set matches the cu13 build context and accepted manual-rebuild boundary..github/workflows/_build-pr-ci-image.yml: verify the captured live-label decision across success, failure, and rerun paths..github/workflows/pr-test.yml: verifytag_availableselects a reused PR image without changing fork or non-Docker behavior.