Got "invalid response status 403" when building docker image

[MegaRedHand]

What's your CLI version?

1.4.3

Description & steps to reproduce

On some machines, using ImageBuildSpec to build images fails with the following error:

There was an error validating Starlark code
Error while validating final environment of script
	Caused by: Failed to build the required image 'foo'.
	Caused by: Image build for 'foo' failed with the following output:
{"id":"moby.buildkit.trace","aux":"Cm8KR3NoYTI1NjpkOGU1MDU3NjEyOWVlYjZjMDg2OGNiZjhhNjY2M2I4ZjZiZjYyYjI2MzRlNTI3M2VmN2Y5NzU3ODgxYWQ0MDdkGiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQ="}
{"id":"moby.buildkit.trace","aux":"Cn0KR3NoYTI1NjpkOGU1MDU3NjEyOWVlYjZjMDg2OGNiZjhhNjY2M2I4ZjZiZjYyYjI2MzRlNTI3M2VmN2Y5NzU3ODgxYWQ0MDdkGiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQqDAjkoZa7BhCZ+YmOAg=="}
{"id":"moby.buildkit.trace","aux":"CqgBCkdzaGEyNTY6ZDhlNTA1NzYxMjllZWI2YzA4NjhjYmY4YTY2NjNiOGY2YmY2MmIyNjM0ZTUyNzNlZjdmOTc1Nzg4MWFkNDA3ZBokW2ludGVybmFsXSBsb2FkIHJlbW90ZSBidWlsZCBjb250ZXh0KgwI5KGWuwYQmfmJjgIyDAjkoZa7BhDWv4GsAjobaW52YWxpZCByZXNwb25zZSBzdGF0dXMgNDAz"}
{"errorDetail":{"message":"failed to read downloaded context: failed to load cache key: invalid response status 403"},"error":"failed to read downloaded context: failed to load cache key: invalid response status 403"}
 --- at /home/circleci/project/container-engine-lib/lib/backend_impls/docker/docker_manager/docker_manager.go:1478 (DockerManager.BuildImage) ---
Container images used in this run:
> foo - locally built
Error encountered running Starlark code.

This behaviour can be reproduced with the following script:

#!/usr/bin/env bash

package_dir=error403

mkdir -p $package_dir

# Any dockerfile will do
cat <<EOF > $package_dir/Dockerfile
FROM badouralix/curl-jq:latest
EOF

# Needed for this to be a valid Kurtosis package
echo 'name: "github.com/foo/bar"' > $package_dir/kurtosis.yml

# This is the script that triggers the error
cat <<EOF > $package_dir/main.star
def run(plan, args={}):
    plan.add_service(
        "foo",
        config=ServiceConfig(
            image=ImageBuildSpec(
                image_name="foo",
                build_context_dir=".",
            ),
            entrypoint=["sleep", "99999"],
            description="Triggering 403 error",
        ),
    )
EOF

echo "Created Kurtosis package"

# This fails
kurtosis run --enclave error403 $package_dir

echo "Ran Kurtosis package. Cleaning up..."

# Clean up enclave, images, and package
kurtosis enclave rm -f error403
docker rmi foo
rm -rf error403

Building the image manually with docker, which causes the build to be skipped in Kurtosis, avoids the issue.

Desired behavior

I expected the image to be built

What is the severity of this bug?

Critical; I am blocked and Kurtosis is unusable for me because of this bug.

What area of the product does this pertain to?

CLI: the Command Line Interface

[tedim52]

Hey @MegaRedHand ! What machine(arch/os) are you running on?

[MegaRedHand]

It works on my M3 MacBook Pro, yet fails on another M3 MacBook Pro we tried it on. Haven't tried it on non-Darwin machines.

[samlaf]

Also getting this error on macbook pro m1
Darwin Samuels-MacBook-Pro.local 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:15 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6000 arm64

[MegaRedHand]

Might be related to moby/moby#47717

[MegaRedHand]

Turning off the Docker Desktop option "Use containerd for pulling and storing images" seems to fix this, as per pulumi/pulumi-docker#967

[samlaf]

Yep that fixed it for me. Would be nice if there was a way to disable this feature on a per-project basis though.

[tedim52]

Thanks for the find @MegaRedHand - I had that option off and image builds were working but when I turn it on it stops working.