Repository navigation
fix(docker): decouple deps-production-base from installed-deps - #2017
Conversation
Fixes the real root cause behind issue #2015 (and #2002): installed-deps is concurrently extended by a second live branch (source-copied, via FROM inheritance) for the rest of the build. deps-production-base's three COPY --from=installed-deps steps were reading that stage while it was still being written to elsewhere in the same build DAG, which hit a BuildKit race resolving the cross-stage ref -- "failed to calculate checksum of ref ...: not found" -- non-deterministically but at a very high rate on GH Actions runners (10/10 recent attempts across #1956 and #1952, even fully cache-free). Retries and cache tuning (#2013, #2016) didn't fix it because it isn't a caching issue. Running npm ci directly in deps-production-base (which already branches straight from node:\${NODE_VERSION}, never touching installed-deps or build) fully decouples this lineage: no more cross-stage read of a stage still being concurrently written to. Cost is one extra @discordjs/opus native compile, once per build (this stage is shared by both deps-production-bot and deps-production-backend), paid once instead of the 3-5 min routinely wasted on exhausted retries. Not locally build-verified end-to-end: native C compilation under QEMU cross-arch emulation (amd64 on this arm64 Mac, via colima) segfaults GCC independent of this change (cc: internal compiler error: Segmentation fault signal terminated program cc1) -- a known QEMU/cross-compile instability class, not present on real amd64 hardware. Relying on CI (real amd64 runners) to validate.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Dockerfile now installs production dependencies in the shared production base with cached ChangesProduction dependency installation
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The PR separates production dependency installation from the shared build stage to avoid the reported Docker build race, and no actionable merge-blocking risk remains beyond normal CI and review checks. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Failed to generate code suggestions for PR |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
No issues found across 1 file
Auto-approved: Focused Docker build fix that eliminates a cross-stage COPY race by running an independent npm ci; no runtime, contract, or security behavior change, and the added build cost (one extra native compile) is minor and justified.
Re-trigger cubic
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This pull request modifies the Docker build to replace the COPY --from=installed-deps steps in the deps-production-base stage with an independent npm ci invocation (using a cache mount), and removes the per-package node_modules copies from the deps-production-bot and deps-production-backend stages. The stated intent is to decouple this stage lineage from the concurrently-written installed-deps stage to avoid a BuildKit cross-stage race (issue #2015). The changes touch only the dependency-installation portion of the multi-stage Dockerfile.
No blocking issues surfaced.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 0 functions depend on the 0 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
|
Even fully isolated (no concurrent PR builds, no cache at all), the docker-build backend leg failed 5/5 attempts on both #1956 and #1952 with the same error the Dockerfile fix in #2017 partially addressed: failed to calculate checksum of ref ...: not found That fix (decoupling deps-production-base from installed-deps) validated clean once but didn't fully eliminate the race -- the same signature reappeared on a different COPY --from step (production-backend <- deps-production-backend), with the unrelated build stage's own npm ci still running concurrently in the log at the moment of failure. This is a BuildKit solver-level race under its default concurrent stage scheduling, confirmed independent of caching (#2013) and independent of cross-job/cross-PR concurrency (reproduces running fully alone). buildkitd-config-inline sets max-parallelism=1 on the ephemeral docker-container builder, forcing one stage operation at a time. Trades some build wall-clock for eliminating the race outright instead of retrying around it.
…#2018) ## Context Follow-up to #2017. That fix (decoupling `deps-production-base` from `installed-deps`) validated clean once (4/4 services, attempt 1, no retries) but didn't fully eliminate the race — rebuilding #1956 and #1952 afterward, both failed 5/5 attempts again with the identical error, now on a different `COPY --from` step (`production-backend` reading `deps-production-backend`), with the `build` stage's own unrelated `npm ci` still running concurrently in the log at the moment of failure. Ruled out as causes: gha cache (#2013 — fails identically fully cache-free), cross-job/cross-PR concurrency (fails identically running fully alone on an isolated rerun). This is a BuildKit solver-level race under its default concurrent stage scheduling — confirmed via Docker's own docs that `max-parallelism` is exactly the documented knob for "particularly useful for low-powered machines" style solver concurrency issues. ## Fix `buildkitd-config-inline` on the `docker/setup-buildx-action` step sets `max-parallelism = 1` (both `[worker.oci]` and `[worker.containerd]`, covering whichever worker the image uses), forcing BuildKit to execute one stage operation at a time instead of racing multiple branches concurrently. Trade-off: slower builds (no more parallel stage execution) in exchange for actually eliminating the race instead of retrying around it. Can tune back up (e.g. `max-parallelism = 2`) later if this proves too conservative once we have clean data. ## Test plan - [ ] This PR's own docker-build matrix passes on attempt 1 (no retries needed) — the real signal that the race is gone - [ ] Note build duration vs previous runs to gauge the serialization cost <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Serializes Docker BuildKit stage scheduling in CI and production publishing to eliminate intermittent "failed to calculate checksum of ref ...: not found" during COPY --from. Previously stages ran concurrently; now both workflows set max-parallelism=1, trading some build speed for reliability. **Details** - Configure `docker/setup-buildx-action` with `buildkitd-config-inline` setting `[worker.oci]` and `[worker.containerd]` `max-parallelism = 1` in `.github/actions/docker-build-service/action.yml` and `.github/workflows/docker-publish.yml`. - Update `.github/workflows/ci.yml` path filter to include `.github/actions/docker-build-service/` so docker-build runs when its composite action changes. - Clarify Dockerfile header on reproducing vs suppressing the race; no functional Dockerfile changes. <sup>Written for commit 5c23bea. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/LucasSantana-Dev/Lucky/pull/2018?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Docker image build reliability by serializing build stages, helping prevent intermittent checksum-related failures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
🤖 I have created a release *beep* *boop* --- <details><summary>2.39.4</summary> ## [2.39.4](v2.39.3...v2.39.4) (2026-08-14) ### Bug Fixes * **ci:** bump docker-build retries from 2 to 4 (5 attempts total) ([#2016](#2016)) ([25af082](25af082)) * **ci:** disable cache-to (not just cache-from) on docker-build retries ([#2013](#2013)) ([608f1d5](608f1d5)) * **ci:** serialize buildkit stage execution to kill the checksum race ([#2018](#2018)) ([ec4b5fd](ec4b5fd)) * **docker:** decouple deps-production-base from installed-deps ([#2017](#2017)) ([cfd3a5e](cfd3a5e)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Release** * Updated the application version to **2.39.4**. * Documented improvements to CI retries, Docker caching, BuildKit handling, and production dependency installation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img](https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a){kind=link}
1 participant
Root cause (issue #2015)
`installed-deps` (an empty checkpoint = `FROM build AS installed-deps`) is consumed by two different lineages within a single `docker buildx build` invocation for any `production-*` target:
Both lineages read/extend `installed-deps` concurrently — BuildKit parallelizes independent branches of the stage DAG by default. The `COPY --from` reads in branch 2 race against branch 1 still actively extending the same stage, and intermittently fail:
```
ERROR: failed to calculate checksum of ref ...: "/app/packages/backend/node_modules": not found
```
Confirmed not a caching issue: reproduced 10/10 recent attempts across #1956 and #1952 with the gha cache fully disabled (#2013), and retries alone don't reliably dodge it even with 5 attempts (#2016).
Fix
`deps-production-base` already branches straight from `node:${NODE_VERSION}` (never touches `installed-deps` or `build`). Give it its own `npm ci` instead of copying node_modules out of `installed-deps` — this fully removes the cross-stage read of a stage still being concurrently written to. `deps-production-bot`/`deps-production-backend` no longer need any `COPY --from=installed-deps` at all (npm workspaces already installs every workspace's node_modules from the root `npm ci`).
Cost: one extra `@discordjs/opus` native compile, paid once per build (this stage is shared by both bot and backend production targets) — versus the 3-5 min routinely wasted on exhausted retries recently.
Verification
Not locally build-verified end-to-end — native C compilation under QEMU cross-arch emulation (amd64 on an arm64 Mac via colima) segfaults GCC independent of this change (`cc: internal compiler error: Segmentation fault signal terminated program cc1`), a known QEMU/cross-compile instability class not present on real amd64 hardware. Relying on this PR's own CI (real amd64 runners, matching production) to validate.
Test plan
Summary by cubic
Decouples
deps-production-basefrominstalled-depsby runningnpm ciinstead of copyingnode_modules. This removes a BuildKit race that intermittently failed production builds with checksum-not-found errors.COPY --from=installed-depswithnpm ci(with cache mount) indeps-production-base; removes the per-packageCOPY --from=installed-depsindeps-production-botanddeps-production-backend; keepsnpm prune --omit=dev.installed-depswas read while another branch still extended it, triggering parallel BuildKit races (“failed to calculate checksum of ref …: not found”).@discordjs/opusnative compile once per build; no runtime changes; workspacenpm cistill installs all packagenode_modules.Written for commit 1b1a877. Summary will update on new commits.
Summary by CodeRabbit