Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-remount-fs.service fail with composefs enabled #3193

Open
karuboniru opened this issue Feb 22, 2024 · 7 comments
Open

systemd-remount-fs.service fail with composefs enabled #3193

karuboniru opened this issue Feb 22, 2024 · 7 comments
Labels
area/composefs Issues related to composefs difficulty/medium medium complexity/difficutly issue reward/small This is a minor fix or cleanup triaged This issue has been evaluated and is valid

Comments

@karuboniru
Copy link

  • Enable composefs by adding following lines to ostree-prepare-root.conf:
[composefs]
enabled=yes
  • Apply image and reboot
  • Check systemctl status systemd-remount-fs.service

Result:

2月 22 13:28:48 karubook systemd-remount-fs[19855]: mount: /: mount point not mounted or bad option.
2月 22 13:28:48 karubook systemd-remount-fs[19855]:        dmesg(1) may have more information after failed mount system call.
2月 22 13:28:48 karubook systemd-remount-fs[19854]: /usr/bin/mount for / exited with exit status 32.
2月 22 13:28:48 karubook systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE

This might be a systemd bug that tried to re-mount the overlay on / type overlay (ro,relatime,seclabel,lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on). If you can comfirm this I can file an another issue on systemd repo.

@karuboniru
Copy link
Author

Cleaning the / entry in /etc/fstab can workaround this, while this introduced another issue where during shutdown there is a [Failed] Failed unmounting sysroot.mount - /sysroot without any information being given in journal.

@cgwalters
Copy link
Member

What's in your /etc/fstab?

@cgwalters cgwalters added needinfo This issue needs more information from the reporter area/composefs Issues related to composefs labels Feb 23, 2024
@karuboniru
Copy link
Author

# UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /                     btrfs   subvol=root,compress=zstd:1,noatime,user_subvol_rm_allowed,ro 0 0
UUID=f35b55c6-f8c3-4279-be97-8811abda483a /boot                   ext4    defaults        1 2
UUID=6907-17CA          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /var                    btrfs   subvol=var,compress=zstd:1,noatime,user_subvol_rm_allowed 0 0
UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 /var/home               btrfs   subvol=home,user_subvol_rm_allowed,noatime     0 0

I understand that the remount failed due to / no longer being a "naked btrfs mount` but now an overlayfs. So I commented it out.

@cgwalters
Copy link
Member

The general fix here is to use rw in your kernel commandline. Hmm, actually you have ro in your mount options...is that intentional? What's in your /proc/cmdline? Was your rootfs being mounted rw before? You have writable subvolumes, but still...

@karuboniru
Copy link
Author

actually you have ro in your mount options...is that intentional

Sorry, that should be added during the process when I am trying to workaround the issue myself. I first tried to add the ro and it did not work then I commented the whole line out.

The cmdline is

BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-0ce8c3b357a4e48a8c5b1617d75fbdb00219f87679fc273acb00dd831cddd07d/vmlinuz-6.8.0-0.rc5.41.fc40.x86_64 rhgb quiet root=UUID=1eef9f42-40e3-4bd8-ae20-e9f2325f8b52 rootflags=subvol=root rw ostree=/ostree/boot.0/fedora/0ce8c3b357a4e48a8c5b1617d75fbdb00219f87679fc273acb00dd831cddd07d/0

@cgwalters
Copy link
Member

I think we'll need to patch systemd for this most likely; there's a bit of a conflict here started from the basic fact that composefs is inherently an overlayfs on top of the underlying real root, and systemd-remount-fs is unaware of this.

Let's take as a prerequisite that we have an elegant way for external tooling to "peel" the overlayfs and find the backing filesystem - singular. In theory a composefs (as it's an overlayfs) can span multiple filesystems, but I struggle to think of a realistic use case for that - at least for the root filesystem.

The systemd volatile root logic has an ad-hoc symlink it creates...would be nicer to be able to attach this as in-band metadata on the mount somehow. But I guess for our use case, similar to the systemd one we're just handling the single root filesystem so a single well-known symlink is probably OK for now.

We then have a tension around the writability of the underlying real root vs the composefs. This issue relates to #3177 a lot.

I'd still take the basic stance that it just makes sense to mount the real root writable (if that's the intent) from the initramfs from the start.

Probably the most practical change is for systemd-remount-fs to gain some logic for detecting that the root it sees is an overlayfs, and if the backing fileystem matches that defined in /etc/fstab, it can handle remounting that instead. And in the common/expected case, the writability and mount options should match and it should be a no-op.

@cgwalters cgwalters added difficulty/medium medium complexity/difficutly issue triaged This issue has been evaluated and is valid reward/small This is a minor fix or cleanup and removed needinfo This issue needs more information from the reporter labels Feb 26, 2024
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
@cgwalters
Copy link
Member

Some code for this in containers/bootc#417 - but as I said there it's really papering over work we need to do probably in both anaconda and systemd.

cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 22, 2024
This is a giant and hacky workaround for
ostreedev/ostree#3193

The better real fix is probably in either systemd or anaconda
(more realistically both) but let's paper over things here for now.

Having code to run as a generator will likely be useful in the
future anyways.

Signed-off-by: Colin Walters <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/composefs Issues related to composefs difficulty/medium medium complexity/difficutly issue reward/small This is a minor fix or cleanup triaged This issue has been evaluated and is valid
Projects
None yet
Development

No branches or pull requests

2 participants