Skip to content

fix(ci): extract node_modules before per-workspace build steps run - #2006

Merged
LucasSantana-Dev merged 11 commits into
mainfrom
fix/checkpoint-node-modules-before-build
Aug 12, 2026
Merged

LucasSantana-Dev merged 11 commits into
mainfrom
fix/checkpoint-node-modules-before-build

Conversation

@LucasSantana-Dev

@LucasSantana-Dev LucasSantana-Dev commented Aug 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Follow-up to #2005. That PR split `deps-production` per production target, which fixed `Build — bot` (it no longer needs the backend build step to run at all, since nothing bot's target consumes depends on it) but `Build — backend` kept failing identically — confirming the bug tracks the backend build step's execution, not which node_modules directory gets copied.

Root cause

`COPY --from=` always waits for that stage's LAST instruction to complete, even when the specific path being copied was already final much earlier. `node_modules` is fully installed by `npm ci`, long before any per-workspace build script runs — none of those scripts write into `node_modules`. But because `packages/backend/node_modules` is copied from the `build` stage (whose last instruction is `RUN npm run build --workspace=packages/backend`), every COPY of it implicitly depends on that RUN completing — and something about that specific RUN step intermittently corrupts BuildKit's cache-key resolution for the immediately-following COPY.

Change

Adds an `installed-deps` checkpoint stage right after `npm ci`, before any per-workspace `RUN` build step. `deps-production-base`/`-bot`/`-backend` now copy `node_modules` from this checkpoint instead of the final `build` stage — fully decoupling the node_modules COPY from whatever the backend build RUN step does. Behavior-neutral: same files, same content, just sourced from an earlier point in the same build graph.

Test plan


Summary by cubic

Decouples Docker COPY sources from later RUNs to avoid intermittent BuildKit checksum failures. Previously production images copied from a monolithic build stage tied to per-workspace builds; now each COPY pulls from the earliest stage that produces the path. Behavior-neutral; still compiles @discordjs/opus once.

  • Add an empty installed-deps checkpoint right after npm ci; deps-production-* now copy each workspace’s node_modules from it.
  • Introduce source-copied (source + prisma generate) and split builds into build-shared → build-bot, build-backend, build-frontend; copy shared/prisma/dist from these stages instead of build.
  • Make build-frontend inherit from build-shared so @lucky/shared/constants resolves; copy @prisma engines and prisma from source-copied.
  • Copy packages/shared/src/generated from source-copied (where prisma generate writes it) instead of build-shared.

Written for commit bed52ee. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Refactor
    • Improved the application build process with more efficient, parallelizable stages.
    • Streamlined production packaging for frontend, backend, bot, and database components.
    • Preserved existing dependency-pruning behavior.

#2005 split deps-production per target, which fixed Build — bot (it
no longer needs the backend build step to run at all) but Build —
backend kept failing identically: failed to calculate checksum of
ref ...: '/app/packages/backend/node_modules' not found, right
after RUN npm run build --workspace=packages/backend completes.

COPY --from=stage always waits for that stage's LAST instruction,
even when the specific path being copied was already final much
earlier (node_modules is fully installed by npm ci, long before any
per-workspace build script runs, none of those scripts write into
node_modules). Add an installed-deps checkpoint stage right after
npm ci, before any per-workspace RUN build step, and copy node_modules
from there instead of the final build stage. Decouples the node_modules
COPY from whatever intermittently corrupts BuildKit's cache-key
resolution around the backend build RUN step.
@LucasSantana-Dev
LucasSantana-Dev enabled auto-merge (squash) August 12, 2026 05:28
@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 87141f5e-5998-4bd8-8ff4-5850e1000765

📥 Commits

Reviewing files that changed from the base of the PR and between 1fb0dee and ebd46de.

📒 Files selected for processing (1)
  • Dockerfile

📝 Walkthrough

Walkthrough

The Dockerfile separates dependency installation, source copying, Prisma generation, and workspace builds into checkpointed stages. Production bot and backend images now copy dependencies and runtime artifacts from these dedicated stages.

Changes

Docker build updates

Layer / File(s) Summary
Separated build checkpoints
Dockerfile
The Dockerfile adds installed-deps, source-copied, build-shared, build-bot, and build-backend stages. The frontend build now branches from build-shared.
Production dependency consumers
Dockerfile
Production bot and backend stages copy dependencies from installed-deps. Bot pruning remains unchanged.
Runtime artifact sources
Dockerfile
Bot and backend runtime stages copy compiled outputs and Prisma engines from dedicated shared, bot, backend, and source stages.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: ci

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the dependency checkpoint change and its purpose within the Docker build refactor.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/checkpoint-node-modules-before-build

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

Failed to generate code suggestions for PR

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR introduces a new installed-deps Docker build stage that is defined immediately after the npm ci step but before any per-workspace build commands run. The deps-production-* stages are then updated to copy their node_modules directories from this new installed-deps stage rather than from the later build stage. The stated intent is to decouple the COPY --from cache-key resolution from the per-workspace build RUN instructions to avoid intermittent BuildKit checksum errors. Surface area is limited to the Dockerfile — one new stage alias and four COPY --from source changes, plus an explanatory comment.

No blocking issues surfaced. 1 lower-confidence candidate did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread Dockerfile
LucasSantana-Dev added a commit that referenced this pull request Aug 12, 2026
## Summary
- Adds a second retry attempt (three total) to the docker-build matrix
job for bot/backend/frontend/nginx.

## Why
The failure isn't a one-off blip: BuildKit intermittently fails to
resolve a COPY --from=stage reference (failed to calculate checksum of
ref ...: not found), and which specific path it hits varies run to run
(node_modules, dist, prisma, generated all reproduced across
#2005/#2006). Disk pressure, GHA cache staleness, buildx version, and
Dockerfile stage structure were all ruled out with direct evidence. This
reads as upstream BuildKit/runner flakiness with a high enough
per-attempt failure rate that a single retry isn't reliably enough
headroom.

## Test plan
- [ ] CI docker-build matrix passes (verifies the new retry step
syntax/behavior end to end)

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Add a second retry to the docker-build matrix job (three attempts total)
for `bot`, `backend`, `frontend`, and `nginx` to mitigate intermittent
BuildKit COPY-from failures and stabilize CI.

<sup>Written for commit 9e0caf0.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/LucasSantana-Dev/Lucky/pull/2007?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR introduces a new installed-deps build stage in the Dockerfile, placed immediately after the npm ci step but before any per-workspace build commands run. The deps-production-* stages are updated to copy node_modules directories from this new checkpoint stage instead of from the later build stage. The change is intended to decouple the COPY --from cache-key resolution from the per-workspace build steps, per an intermittent BuildKit checksum error described in the added comment.

No blocking issues surfaced. 1 lower-confidence candidate did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

LucasSantana-Dev added a commit that referenced this pull request Aug 12, 2026
## Summary
- Skip the gha layer cache import on docker-build retry attempts.

## Why
#2006's own CI run gave the missing piece: all three attempts (build,
retry 1, retry 2) failed identically within the same job, each time on
the exact same buildx ref hash across multiple unrelated COPY paths
(packages/shared/src/generated, packages/shared/dist, packages/bot/dist,
prisma). cache-from is scoped identically (type=gha,scope=<service>)
across every attempt in a job, so a retry that reimports a stale or
corrupted gha cache entry fails identically instead of getting a fresh
build. That's why the retry-count fix in #2007 did not save #2006's run.

Adds a `use-cache` input to `docker-build-service` (default true) and
sets it to false on both retry steps in ci.yml, so retries skip
cache-from and build from scratch. cache-to stays on so a successful
retry still writes a fresh cache for the next run.

## Test plan
- [ ] CI docker-build matrix passes on this PR
- [ ] Resync PR 2006 onto main once this merges and confirm its Build -
bot / Build - backend go green

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Skip importing the GHA layer cache on docker-build retries to prevent
repeated BuildKit failures. Retries now build fresh while still writing
a new cache on success.

- **Bug Fixes**
- Added `use-cache` input to `./.github/actions/docker-build-service`
(default `true`); set to `false` on retry steps in `ci.yml` to omit
`cache-from`.
- Keeps `cache-to: type=gha,mode=max` so successful retries repopulate
the cache and resolves the "failed to calculate checksum of ref ...: not
found" errors from stale `type=gha,scope=<service>` entries.

<sup>Written for commit aee532a.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/LucasSantana-Dev/Lucky/pull/2008?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR introduces a new installed-deps build stage in the Dockerfile, positioned immediately after npm ci but before any per-workspace build steps run. The deps-production-* stages are then updated to copy their node_modules directories from this new checkpoint stage rather than from the final build stage. An extensive comment explains the motivation: decoupling the COPY of node_modules from the later build steps to avoid an intermittent BuildKit cache-key/checksum failure.

No blocking issues surfaced. 1 lower-confidence candidate did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

The checkpoint-stage extraction in this PR moved build:shared out of the
build stage and into installed-deps, but left build-frontend branching
from build directly - before shared gets built. Frontend imports
@lucky/shared/constants, so its tsc build failed with
error TS2307: Cannot find module '@lucky/shared/constants'.

Confirmed via CI: this PR's own Build - frontend job failed
deterministically (3/3 attempts, identical TS2307 errors) once resynced
onto main with the retry/cache-bypass mitigations - a real regression,
not the intermittent BuildKit checksum flake this PR targets.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Dockerfile`:
- Around line 92-93: Restructure the Dockerfile stages around installed-deps:
keep dependency installation in installed-deps, add a workspace-build stage
based on it containing the workspace COPY and build commands, and make
build-frontend inherit from workspace-build. Update all artifact COPY sources to
use workspace-build instead of build so dist, generated sources, and /app/prisma
come from the stage that creates them.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: f38e0f28-38c3-4d8f-a88a-7012b13a8bfb

📥 Commits

Reviewing files that changed from the base of the PR and between 89dbb90 and 3ce976e.

📒 Files selected for processing (1)
  • Dockerfile

Comment thread Dockerfile

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR restructures the Dockerfile by introducing a new installed-deps stage that acts as a checkpoint immediately after npm ci completes but before any per-workspace build steps run. The build-frontend stage and the various deps-production-* stages are updated to copy node_modules from this new installed-deps stage rather than from the later build stage. The stated intent (per the added comments) is to decouple the node_modules copies from the build stage's later RUN instructions to avoid an intermittent BuildKit checksum resolution error. The changes are confined to the Dockerfile: adding one new stage and repointing several COPY --from= and FROM references.

No blocking issues surfaced.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic

Comment thread Dockerfile Outdated

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.


Graphify review — findings

This PR restructures the multi-stage Dockerfile by introducing a new installed-deps stage that is checkpointed immediately after npm ci completes but before any per-workspace build scripts run. The build-frontend stage and the three deps-production-* stages are updated to copy node_modules from installed-deps rather than from the later build stage. The stated intent is to decouple the node_modules copies from the per-workspace build RUN steps to avoid intermittent BuildKit cache-key/checksum errors, with the author asserting no functional change.

Worth a look

  • Frontend build stage no longer contains built shared package — Dockerfile:112 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

cache-from alone disabled (confirmed via #2008) did not fix the
persistent Build - backend flake - reruns still fail deterministically
even with a completely fresh, no-cache-from build. cache-to with
mode=max exports every intermediate layer progressively as the build
runs; a later stage's COPY --from can plausibly race against that same
layer still being finalized for export. Disable cache-to too on
retries so they run as a plain uncached build with no export pressure.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR restructures the Docker build to work around an intermittent BuildKit "failed to calculate checksum" error. It introduces a new installed-deps stage in the Dockerfile that acts as a checkpoint right after npm ci (before per-workspace builds), and repoints the deps-production-* stages and the build-frontend stage to copy node_modules from this new stage instead of the build stage. It also changes the CI docker-build-service action so that cache-to (GHA cache export) is now gated on the use-cache input, meaning it gets disabled alongside cache-from on retries, along with updated explanatory comments. Surface area: one GitHub Actions composite action config and the root Dockerfile (stage graph and inter-stage COPY --from sources).

No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic

Research finding: moby/buildkit v0.32.0 released 2026-08-04, days before
this repo's Build - backend job started hitting a sustained, high-rate
'failed to calculate checksum of ref ...: not found' failure with no
code changes on our side. Multiple independent upstream reports of this
exact error (moby/buildkit#4128, docker/build-push-action#910/#934/#935)
trace to BuildKit-internal snapshot/cache-key resolution bugs, and #910
was resolved for its reporter specifically by pinning BuildKit to an
older build via driver-opts. setup-buildx-action's docker-container
driver otherwise floats to whatever BuildKit tag buildx currently
bundles - pin to v0.31.0 (released 2026-06-17, two months of production
mileage before v0.32.0) to sidestep a possible new-release regression.
…tries

BuildKit v0.31.0 daemon pin (previous commit) was confirmed applied
(builder log showed the pinned image pulled and in use) but the
"failed to calculate checksum of ref ...: not found" flake still hit
on 3/3 retries with an identical signature. Disproven.

Switch retries to the classic `docker` buildx driver instead of the
default docker-container driver. The bug lives in BuildKit's own
snapshot/cache-key resolution inside that separate container's
snapshotter (moby/buildkit#4128, open upstream since 2023) - the
docker driver uses a different snapshotter entirely, sidestepping the
code path rather than hoping a different BuildKit build avoids it.
Retries already forgo gha layer caching (use-cache: false), which the
docker driver doesn't support anyway, so nothing extra is given up.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR targets an intermittent BuildKit "failed to calculate checksum of ref ...: not found" error in the Docker build CI pipeline through several changes. In the composite build action, it adds a new driver input (defaulting to docker-container) that gets wired into setup-buildx-action, and changes use-cache: false so it now disables the gha cache-to export in addition to cache-from. The CI workflow's retry steps now pass driver: docker alongside the existing use-cache: 'false'. In the Dockerfile, it introduces a new installed-deps stage right after npm ci, and repoints the deps-production-* stages and the build-frontend stage to copy from installed-deps instead of the build stage. Extensive comments are added throughout documenting the investigation history and rationale for each lever. The surface area is limited to CI build configuration and Docker stage sourcing; I'm not assessing whether these changes resolve the underlying flake.

No blocking issues surfaced.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread .github/actions/docker-build-service/action.yml Outdated
…o main's retry logic

Both tested and disproven on this branch:
- driver-opts pin to moby/buildkit:v0.31.0 - confirmed applied via
  builder log, identical failure signature on 3/3 attempts
- driver: docker fallback on retries - identical failure signature on
  both retry attempts within the same run, same ref hash pattern as
  the docker-container attempts

The identical error recurring under a different buildx driver (docker
vs docker-container, different snapshotter backends entirely) points
at BuildKit's core LLB cache-key solver, not the container snapshotter
specifically. No further driver/version lever is left to try from this
angle. Falling back to plain retry-until-luck, same as main.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.


Graphify review — findings

This PR refactors the Dockerfile to introduce a new installed-deps build stage that acts as a checkpoint immediately after npm ci completes but before any per-workspace build steps run. The deps-production-* stages and the build-frontend stage are updated to copy node_modules from (and branch from) this new stage instead of the later build stage. The stated intent is to decouple COPY --from operations from build steps to avoid an intermittent BuildKit cache-key/checksum resolution failure, while keeping the frontend build downstream of the build:shared step it depends on.

Worth a look

  • Frontend build stage no longer depends on shared build output — Dockerfile:110 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • Frontend build stage no longer includes built shared package — Dockerfile:110 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/actions/docker-build-service/action.yml">

<violation number="1">
P2: When a retry passes `use-cache: 'false'`, this line still exports `mode=max` to the GHA cache instead of bypassing the cache entirely. Gate `cache-to` with `inputs.use-cache` too, so retries do not repopulate or contend with the stale service cache.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

@LucasSantana-Dev

Copy link
Copy Markdown
Owner Author

The node_modules decoupling didn't address the failures actually being hit

Since this PR's checkpoint-stage fix landed, I've been testing further mitigations on this branch (BuildKit daemon version pin, buildx driver fallback — both disproven, see #2002). Looking closer at the actual failure logs from every attempt on this branch (not just this PR's premise): the failing `COPY --from` paths are consistently

```
"/app/packages/shared/src/generated": not found
"/app/packages/shared/dist": not found
"/app/packages/backend/dist": not found
"/app/prisma": not found
```

None of these is `node_modules`. They're build outputs — copied from the `build` stage's `deps-production-*` COPY instructions, which necessarily run after the per-workspace `RUN npm run build` steps (you can't decouple a build output from the RUN that produces it the way `installed-deps` decouples node_modules, which exists before any RUN).

This PR's root-cause theory ("RUN npm run build --workspace=packages/backend intermittently corrupts BuildKit's cache-key resolution for the immediately-following COPY") predicted the node_modules COPY was the affected one — that's now decoupled and no longer implicated in any of this session's ~10 build attempts on this branch. But the dist/generated/prisma COPYs, which still directly follow build RUN steps in the `build` stage, are exactly what's failing every time.

This matches the upstream bug's shape (moby/buildkit#4128: COPY --from referencing a stage whose last instruction was a RUN, resolved after that RUN completes) more precisely than the original theory did — it's not specific to the backend build RUN corrupting a specific later COPY, it's that any COPY --from immediately following any RUN in that stage is at risk, and this Dockerfile still has several (shared build → shared dist/generated COPY, backend build → backend dist COPY, plus the untouched prisma generate step → prisma COPY).

Not attempting a further Dockerfile restructure on this myself right now — decoupling these COPYs would need a checkpoint-per-build-output pattern (one intermediate stage after each RUN, mirroring what installed-deps did for node_modules), which is a real structural change to this Dockerfile and deserves a proper look with fresh eyes rather than another speculative patch stacked onto an already-long investigation. Full history on #2002.

…x inert checkpoint

Two structural fixes to the BuildKit "failed to calculate checksum of
ref ...: not found" flake (moby/buildkit#4128, open upstream since
2023), found after the observed failure paths this session (shared/dist,
shared/src/generated, backend/dist, prisma) didn't match this branch's
existing installed-deps checkpoint, which only decouples node_modules.

1. The existing `installed-deps` checkpoint (from an earlier commit on
   this branch) was actually inert: every instruction between a
   `FROM ... AS x` and the next `FROM` belongs to x's own chain, and
   this branch appended the source COPY + all three workspace builds
   directly onto installed-deps with no intervening FROM. Its
   addressable snapshot silently included everything through the
   backend build anyway - a rename, not a checkpoint. Fixed by giving
   installed-deps a genuinely empty body and moving source COPY +
   prisma generate into a new source-copied stage.

2. bot/backend/frontend builds ran sequentially in one stage, so a
   COPY --from of shared/dist (ready after step 1 of 3) still had to
   wait for backend's build RUN too - COPY --from=<stage> always waits
   for that stage's last instruction, whatever it is. Split into
   build-shared -> {build-bot, build-backend, build-frontend}
   independent forks, each COPY --from now pointing at the earliest
   stage that actually has the needed path. Side benefit: bot/backend/
   frontend can now build in parallel, since none depends on the
   others' output.

Reviewed by an independent critic pass (adversarial, traced every
COPY --from against its source stage, verified WORKDIR propagation
and the empty-stage checkpoint semantics) before push - no correctness
issues found. Still a speculative fix for a bug neither this nor the
critic's pass could fully root-cause; CI is the real test.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.


Graphify review — findings

This PR restructures the Dockerfile's multi-stage build by splitting the previously monolithic build stage into several distinct stages: installed-deps (a checkpoint right after npm ci), source-copied (source + prisma generate), build-shared, and separate build-bot, build-backend, and build-frontend stages that branch off build-shared. The various COPY --from=build references throughout the production/deps stages are updated to point at whichever new stage is the earliest one that produces the needed artifact (e.g. installed-deps for node_modules, build-shared for shared dist, source-copied for prisma). The stated intent (per the extensive added comments) is to decouple COPY --from steps from unrelated downstream RUN instructions to avoid intermittent BuildKit cache-key/checksum failures, and to allow the bot/backend/frontend builds to run in parallel. The reviewer should focus on whether each stage boundary and updated --from target still references a stage where the copied artifact exists.

Worth a look

  • prisma generate output not propagated to production images — Dockerfile:96 · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@github-actions github-actions Bot added size/m and removed size/s labels Aug 12, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread Dockerfile Outdated
…ared

cubic flagged this on the stage-split commit: packages/shared/src/generated
is Prisma's client output (prisma/schema.prisma output =
"../packages/shared/src/generated/prisma"), written by `npx prisma
generate` in source-copied, not by `npm run build:shared` in
build-shared. Sourcing it from build-shared left it coupled to a RUN
it doesn't actually depend on - the exact pattern this whole change
set out to eliminate.
@LucasSantana-Dev

Copy link
Copy Markdown
Owner Author

Fixed per cubic's review: shared/src/generated now sources from source-copied (right after npx prisma generate, which is what actually writes it — prisma/schema.prisma output is ../packages/shared/src/generated/prisma) instead of build-shared.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR restructures the multi-stage Dockerfile by splitting the single monolithic build stage into several smaller stages: installed-deps (checkpoint right after npm ci), source-copied (source COPY + prisma generate), build-shared, and separate build-bot, build-backend, and build-frontend stages that branch off build-shared. The various COPY --from=build references throughout the production stages are correspondingly repointed to whichever new stage last touched the artifact being copied (e.g. node_modules from installed-deps, dist output from build-shared/build-bot/build-backend, generated/prisma files from source-copied). The stated intent, per the added comments, is to decouple unrelated COPY --from steps from later build RUN steps in the same stage (to work around an intermittent BuildKit checksum error) and to allow the per-workspace builds to run in parallel. The changes are confined to Dockerfile; no application code is touched.

No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

@sonarqubecloud

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Auto-approved: Restructures Docker build stages to decouple COPYs from later RUNs, fixing a BuildKit cache bug; behavior-neutral build tooling change with no runtime, security, or data impact visible.

Re-trigger cubic

@LucasSantana-Dev
LucasSantana-Dev merged commit ae577d1 into main Aug 12, 2026
44 checks passed
@LucasSantana-Dev
LucasSantana-Dev deleted the fix/checkpoint-node-modules-before-build branch August 12, 2026 17:22
LucasSantana-Dev added a commit that referenced this pull request Aug 12, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>2.39.3</summary>

##
[2.39.3](v2.39.2...v2.39.3)
(2026-08-12)


### Bug Fixes

* **bot:** expose degraded-extractor signal and honest play error
([#1999](#1999))
([8348bd6](8348bd6))
* **bot:** unlink dead Last.fm sessions on error 9 and notify once
([#1946](#1946))
([d22b8bd](d22b8bd))
* **ci:** add a second retry for docker-build
([#2007](#2007))
([9347c55](9347c55))
* **ci:** extract node_modules before per-workspace build steps run
([#2006](#2006))
([ae577d1](ae577d1))
* **ci:** free disk space before Docker builds to prevent BuildKit GC
race ([#2003](#2003))
([5bcd388](5bcd388))
* **ci:** isolate docker cache scope by branch
([#2012](#2012))
([5d25ec8](5d25ec8))
* **ci:** pin node:24-alpine base image by digest
([#2004](#2004))
([f154bfe](f154bfe))
* **ci:** skip gha cache-from on docker-build retries
([#2008](#2008))
([89dbb90](89dbb90))
* **ci:** split deps-production per production target
([#2005](#2005))
([43aacce](43aacce))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant