authfile paths can't be absolute

[prydom]

I get the following error when attempting to use bootc to upgrade my laptop.

$ sudo bootc upgrade
ERROR Upgrading: Pulling: Creating importer: a path led outside of the filesystem

This same error does not occur when using rpm-ostree. Additional layers are successfully deployed.

$ rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:snapshot
No upgrade available.

I am using the current Fedora rawhide packages for both tools. Both tools seem to use https://github.com/ostreedev/ostree-rs-ext/blob/dc5faade367e8cb20997163a6b56a8da8f1596e5/lib/src/container/store.rs#L451 and the "a path led outside of the filesystem" error seems to trace to the https://github.com/bytecodealliance/cap-std/tree/v3.0.0 crate. This started occurring recently although I can't pinpoint exactly when. This is as far as I've gotten so far with a RCA.

(EDIT: removed irrelevant environmental details.)

Is there any suggested way to continue to debug this error?

[cgwalters]

Is there any suggested way to continue to debug this error?

Try
sudo strace -s 2048 -f --status=failed bootc upgrade

[prydom]

Ah @cgwalters, seems like an issue with resolving a symlink to my registry auth token.
My image is private due to bundling proprietary packages and video codecs.

$ sudo strace -s 2048 -f --status=failed bootc upgrade

[...]
openat2(17, "etc/ostree/auth.json", {flags=O_RDONLY|O_LARGEFILE|O_CLOEXEC, resolve=RESOLVE_NO_MAGICLINKS|RESOLVE_BENEATH}, 24) = -1 EXDEV (Invalid cross-device link)
ERROR Upgrading: Pulling: Creating importer: a path led outside of the filesystem
+++ exited with 1 +++
                               
$ ls -lhA /etc/ostree                                                                    
total 8.0K
drwxr-xr-x. 1 root root 60 Jul 12 12:27 remotes.d
lrwxrwxrwx. 1 root root 25 Jul 12 12:28 auth.json -> /etc/containers/auth.json
-rw-r--r--. 1 root root 22 Jun  6 15:07 prepare-root.conf

Please let me know if you'd like me to contribute a PR or if this is easy enough to fix yourself.

[prydom]

This workaround is sufficient as RESOLVE_BENEATH is the blocker:

$ sudo ln -s ../containers/auth.json auth.json

$ ls -l
total 8
drwxr-xr-x. 1 root root 60 Jul 12 12:27 remotes.d
lrwxrwxrwx. 1 root root 23 Jul 12 12:42 auth.json -> ../containers/auth.json
-rw-r--r--. 1 root root 22 Jun  6 15:07 prepare-root.conf

$ sudo bootc upgrade
No changes in ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:snapshot => sha256:b207ab0a7fc6848e62b9ae6f5d17cead9bd678c6dd963b131620cb96be8a3ae6
No update available.

Would allowing absolute symlink resolution present a (security) risk here? In any case, the error message presented is non-obvious and the original symlink works with rpm-ostree at least for now. The two tools use different crate versions for cap-std, so maybe that behavior will change in future?

[cgwalters]

Yeah, this is a bug. You can work around it by making the symlink relative, not absolute. IOW something like ln -sfr /etc/containers/auth.json /etc/ostree/auth.json or so in the buildsystem (operative bit being -r).

For fixing it in the code...it's a bug in the code around https://github.com/ostreedev/ostree-rs-ext/blob/dc5faade367e8cb20997163a6b56a8da8f1596e5/lib/src/globals.rs#L56 but it's an annoying thing to fix and we are going to randomly hit things related to this until we address it in a more structured fashion. cc bytecodealliance/cap-std#353 (comment)

[cgwalters]

In fact this is a regression from ostreedev/ostree-rs-ext#636 and a bad one because anyone who was using an absolute path is just going to get broken in rpm-ostree too...I will look at this today or Monday.

[prydom]

Thanks for the full RCA!

I don't bundle auth.json in my image so this is something that gets carried forward in the /etc three-way merge.
However this will break my Github Actions workflow as I do rely on symlinks there to login to the ghcr.io registry to push with rpm-ostree compose container-encapsulate. I'll add the -r switch to that step.

[cgwalters]

Where is the symlink coming from? Is it something in prydom/bluebuild-github-action?

[prydom]

No. Just good old manual bootstrapping prior to rebasing from upstream Kinoite or after initial deployment of the image. e.g. Doing the deployment with bootc or in my case by ostree admin init-fs --modern && ostree admin stateroot-init && ostree container image deploy with the appropriate flags.

I usually do the deployment entirely by hand from a recovery image that I build from rawhide - so that I control the bootloader as I use systemd-stub UKIs, systemd-cryptenroll, and setup my own secure boot signing chain.

[prydom]

If you meant before container-encapsulate it's here: https://github.com/prydom/my-ostree-build/blob/0b6573a18f88a48faba9f0c0653ba5cf7caf134e/.github/workflows/build.yml#L65 on line 65.

[cgwalters]

I did:

• Add a RootDirectory API coreos/cap-std-ext#54
• config: Fix absolute symlinks ostreedev/ostree-rs-ext#650

Then we just need to do an ostree-rs-ext release and bump here.