Unable to bind a signed image at install time

[ckyrouac]

When bootc tries to copy the signed image from the host to the install disk it fails with the following error:

ERROR Installing to disk: Pulling from host storage: registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0-13: Subprocess failed: ExitStatus(unix_wait_status(32000))
Getting image source signatures
Checking if image destination supports signatures
Error: Copying this image would require changing layer representation, which we cannot do: "Would invalidate signatures"

Adding "--remove-signatures" to the podman image push command here fixes this error, however the result is an unsigned image. I did some digging through the containers/image code, skopeo docs, and containers-storage.conf docs. It looks like the only way to copy and sign an image is to re-sign the image when copying it, I couldn't find a way to copy a signed image while preserving the signature. I might be missing something though.

[cgwalters]

Hm, only glancing at this one thing I notice is

$ skopeo inspect --raw -n docker://registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0-13 | jq -r .mediaType
application/vnd.docker.distribution.manifest.list.v2+json

It's a legacy media type, not OCI. This is a thing we should start trying to change across Red Hat images.

But indeed there may be a larger conflict here even with OCI that may need fixing.

[Odilhao]

Hey @cgwalters we just stumbled on this while tying to locally build one RHEL 9.4 image.

With a really simple config.toml:

[[blueprint.customizations.user]]
name = "someuser"
password = "somepassword"
key = ""
groups = ["wheel"]

sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v ./config.toml:/root/config.toml:ro -v ./output:/root/output -v /var/lib/containers/storage:/var/lib/containers/storage registry.redhat.io/rhel9/bootc-image-builder:latest --local --type qcow2 --config /root/config.toml registry.redhat.io/rhel9/rhel-bootc:9.4

That fails with:

Checking if image destination supports signatures
time="2024-10-03T20:25:35Z" level=fatal msg="Copying this image would require changing layer representation, which we cannot do: \"Would invalidate signatures\""
Error: tmp-container-deploy-71731344553455: image not known
Traceback (most recent call last):
 File "/run/osbuild/bin/org.osbuild.container-deploy", line 50, in main
   subprocess.run(
 File "/usr/lib64/python3.9/subprocess.py", line 528, in run
   raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'copy', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]1f64fbbed435d0f92755b9e6f6521e8d20c3d00f62cb869774cc6d0aab9bf897', 'containers-storage:tmp-container-deploy-71731344553455']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/run/osbuild/bin/org.osbuild.container-deploy", line 64, in <module>
   r = main(args["inputs"], args["tree"], args["options"])
 File "/run/osbuild/bin/org.osbuild.container-deploy", line 56, in main
   subprocess.run(["cp", "-a", f"{img}/.", f"{tree}/"], check=True)
 File "/usr/lib64/python3.9/contextlib.py", line 532, in __exit__
   raise exc_details[1]
 File "/usr/lib64/python3.9/contextlib.py", line 517, in __exit__
   if cb(*exc_details):
 File "/usr/lib64/python3.9/contextlib.py", line 405, in _exit_wrapper
   callback(*args, **kwds)
 File "/usr/lib64/python3.9/subprocess.py", line 528, in run
   raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'rmi', 'tmp-container-deploy-71731344553455']' returned non-zero exit status 1.

[cgwalters]

Hi @Odilhao that issue is related (in fact ultimately the same I suspect) but practically distinct because the failure there is in bootc-image-builder code.

For reasons I don't understand really (and we're debating in multiple places), bootc-image-builder ends up copying the input container's rootfs and running it via custom tooling instead of just launching it via a standard OCI mechanism (e.g. podman).

[cgwalters]

cc @mvo5 @achilleas-k re ⬆

[achilleas-k]

For reasons I don't understand really (and we're debating in multiple places), bootc-image-builder ends up copying the input container's rootfs and running it via custom tooling instead of just launching it via a standard OCI mechanism (e.g. podman).

There's a couple of things we can simplify in the way we move around the container. We initially read it from the host container store, copy it to an internal store, then mount it so we can copy out the contents and create a build root. The copy into the internal store can probably be skipped, I think it's a leftover from previous workflows that required it.

I still think mounting and copying the container contents out to use its tooling is the right thing to do in osbuild. The alternative, using podman run could work, but it would require running pretty much all osbuild stages through podman. Maybe I'm thinking about this wrong, but what we're talking about here is replacing:

podman mount <container> <mnt>
cp -a <mnt> <build root>
<build root>/truncate ...
<build root>/mkfs.ext4 ...

with

podman run <container> truncate ...
podman run <container> mkfs.ext4 ...

The latter would need a lot of changes in the internals of osbuild, all of which would be to serve image mode and have no value for every other kind of build. Perhaps it's worth considering replacing bubblewrap in osbuild's internals with podman, change the way we set up build roots to be compatible with the bootc world, so that we can reuse this workflow everywehre, but that's a much bigger story and effort than "just launching tooling via a standard OCI mechanism".

[cgwalters]

The alternative, using podman run could work, but it would require running pretty much all osbuild stages through podman.

Can you explain a bit more about why that is? It's not obvious to me why that'd be the case. I think it's more about replacing just that specific bit with podman run (or equivalent) to start.

Anyways just to xref it looks like the bib side of this is now in osbuild/bootc-image-builder#676