v0.33.0 bug: CheckReadAccess fails when using a local run image

[sophiewigmore]

Summary

---

See https://cloud-native.slack.com/archives/C0331B61A1Y/p1708468320706469 for more context.
When upgrading to pack v0.33.0 from v0.32.1 in our stack implementation test suite, we run into the following error during a pack build:

 CheckReadAccess failed for the run image stack-run-845695c9-5147-42a2-8cb3-c0ea0500c6a4:latest, error: HEAD https://index.docker.io/v2/library/stack-run-845695c9-5147-42a2-8cb3-c0ea0500c6a4/manifests/latest: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
        panic: runtime error: index out of range [0] with length 0

We are creating a stack build/run image locally, then creating a builder with those stack images, and then using the builder in a pack build. As a result of the changes in this PR, this no longer works due to issues with accessing the local run image on the docker daemon.

As a workaround, we are able to create a local docker registry, and push the stack images, which helps us to avoid this issue. We would still like to to be able to build and test stack images locally without needing to do this, however.

Reproduction

Steps

1. Install pack v0.33.0
2. Create a stack build/run image locally
3. Create a builder with the local stacks using pack
4. Use the builder in a pack build of a simple app

OR more simply

1. Checkout the Paketo Jammy Base Stack at this commit
2. Run the integration tests via scripts/test.sh which will do the above steps
3. See the failure above in the buildpack_integration_test.go test case

Current behavior

Build failed due to inability to access the local run image

Expected behavior

Successful build

---

Environment

pack info

 ‣ .bin/pack report
Pack:
  Version:  0.33.0+git-e3f8bc2.build-5475
  OS/Arch:  darwin/amd64

Default Lifecycle Version:  0.18.4

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
  default-builder-image = "[REDACTED]"
  experimental = true
  layout-repo-dir = "/Users/swigmore/.pack/layout-repo"

  [[trusted-builders]]
    name = "[REDACTED]"

  [[trusted-builders]]
    name = "[REDACTED]"

  [[trusted-builders]]
    name = "[REDACTED]"

  [[trusted-builders]]
    name = "[REDACTED]"

[natalieparellano]

This is unfortunate. I think the right way to fix this is to make CheckAccess (or something) a method on the ImageFetcher (implemented here). Then we can check the access in the daemon and/or the registry depending on the pull policy.

[jochenr]

I experience the same error.
My expectation was that "--pull-policy=never" would also solve this for the run-image as it did for the builder-image

[jochenr]

Since I could not continue my work because of this issue with the local image from my Podman installtion, I tried to use our private "ghrc.io/my-company..." registry on github as workaround.

Unfortunately this workaround does not work;-(

the

pack build sample-app --path ./my-project --builder ghcr.io/my-company/...  --pull-policy=if-not-present

command gives

CheckReadAccess failed for the run image ghcr.io/.... , error: GET https://ghcr.io/token?scope=repository%3A.....pull&service=ghcr.io: UNAUTHORIZED: authentication required
panic: runtime error: index out of range [0] with length 0

although my local podman login into ghcr.io is working.

is this also a (another) bug?

[jjbustamante]

Hi @jochenr

In theory the workaround is to point to a remote registry, it could be a local v2 registry where you push the image and point to that one, or as you said, using your company private registry. It looks like you don't read access to that remote registry?

service=ghcr.io: UNAUTHORIZED: authentication required
panic: runtime error: index out of range [0] with length 0

I just updated my branch with the latest main, if it is possible for you, could you test the binary from that branch to verify the error is fixed?

[jjbustamante]

Once the CI job finish, the binaries must be available here

[jochenr]

Hi @jjbustamante,

thank you!
I'll give it a try on monday when I'm back in office

[jochenr]

Good Morning @jjbustamante,

good news.
With the very new version of the CI Build you pointed me to, my local builder and runtime images are found.

Thank you!

[jjbustamante]

Thanks!!! @jochenr that's really helpful