Build fails if containerd is used with an untrusted builder

[modulo11]

Summary

Using the containerd backend in Docker together with an untrusted builder fails the build:

ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:6528df63e9cafd1747f04fc57cad7b3910dce6be0e6cf49e8174fb2aeda69ef6" from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:2fc1acc6fd5a97e82e2b780700a2c07e813e84fac1fe5857b42f475e6dc526d0: not found

Docker Desktop enabled the containerd backend for new installations starting with v4.34.0.

Reproduction

Steps

1. Enable containerd backend
2. git clone https://github.com/paketo-buildpacks/samples.git
3. cd samples/nodejs/npm
4. pack --verbose build --clear-cache --builder paketobuildpacks/builder-jammy-base:latest sample-node

Current behavior

Build fails.

Expected behavior

Build succeeds.

Environment

pack info

Pack:
  Version:  0.35.1+git-3a22a7f.build-6099
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.20.0

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12, 0.13

Config:
(no config file found)

docker info

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 27.2.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.655GiB
 Name: docker-desktop
 ID: 1a97794d-938a-4f61-afee-1ce0744f88b0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal

[jjbustamante]

@modulo11

Have you tried this on an intel Macos machine? I think paketobuildpacks/builder-jammy-base builder is not available for ARM yet

[modulo11]

No, I sadly cannot test on Intel. Building works on my machine if I disable containerd, both use AMD64 builder images.

What I noticed is that imgutil uses different code paths when for saving if containerd is enabled or not. I was unsure where to file the issue, but I can open is there as well if you like.

[jjbustamante]

Yeah, if I remember correctly, using containerd has a performance penalty right now, some context here and we have that different path on imgutil, I will try to reproduce it on intel with containerd

[natalieparellano]

It's weird that trusted / untrusted seems to make a difference here. What image are we trying to fetch the base layers for?

[natalieparellano]

FWIW the error message seems to be coming from here: https://github.com/containerd/containerd/blob/61f91b963ef244daec1bda6700fe3f0b1aee50c6/core/images/archive/exporter.go#L335

I haven't found any references to this error in issues in any of buildpacks/moby/containerd. Would take some digging to understand how this could occur.

This reminds me of this issue we encountered when iterating between storage drivers. I know this is unsatisfying, but could you perhaps try clearing your image cache to see if that helps at all?

[natalieparellano]

@modulo11 any further thoughts on this one?

[modulo11]

Not really. After #2266 has been merged, the situation improved a bit for me.