Support/document usage with read-only rootfs on the host

[yangm97]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Currently its not possible to perform even read-only operations (such as podman ps) when the graphroot happens to be in a read-only volume (for instance, when the host / is mounted as ro, either intentionally or because ext4 went "oopsie").

However we already a somewhat working solution for that specific use case: move the graphroot to an ephemeral mount. Assuming you want to use images and volumes stored at the default graphroot (as I did for testing this idea) you could change your config like this:

/etc/containers/storage.conf:

[storage]
graphroot = "/run/containers/storage"

[storage.options]
additionalimagestores = [
  "/var/lib/containers/storage"
]

/etc/containers/containers.conf:

[engine]
volume_path = "/var/lib/containers/storage/volumes"

This will work fine until the next reboot. Since volume metadata seems to be persisted on the boltdb store (and not in the volume_path), podman won't know about their existence once the graphroot is recreated, and trying to recreate those volumes results in an error since f7e72bc: #8254

Error: error creating volume directory "/var/lib/containers/storage/volumes/bla/_data": mkdir /var/lib/containers/storage/volumes/bla/_data: file exists

I don't know if this behavior is intentional or not but I've created a little PR which should revert it:

I know this PR doesn't actually fix the core issue but it should be enough of a fix for when you want:

• images stored in a read-only partition
• user data is in a rw partition
• podman state is as ephemeral as the containers it manages

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

(paste your output here)

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

What error are you seeing when the system is in read/only mode? Overlayfs complaining about not being setup correctly?

[yangm97]

@rhatdan

Error: error opening database /var/lib/containers/storage/libpod/bolt_state.db: open /var/lib/containers/storage/libpod/bolt_state.db: read-only file system

[rhatdan]

@mheon I would figure this would take quite a bit of work to make the database able to handle read/only mode? Basically we are opening the database in read/write mode and the open is failing. If we fell back to read/only mode, then lots of other calls are going to fail.

[mheon]

I don't think we can support r/o - the container state has to be stored somewhere (PID, IP addresses, etc). It may be feasible to store the DB on a tmpfs but we'd lose track of new containers on a reboot.

…

On Sat, Nov 7, 2020, 06:20 Daniel J Walsh ***@***.***> wrote: @mheon <https://github.com/mheon> I would figure this would take quite a bit of work to make the database able to handle read/only mode? Basically we are opening the database in read/write mode and the open is failing. If we fell back to read/only mode, then lots of other calls are going to fail. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8253 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCGZYWN6BDKPLRQCDT3SOUUQFANCNFSM4TL4JGQQ> .

[rhatdan]

I don't think @yangm97 is after full support, it would just like to run the read/only type functions, podman version, podman info, podman ps.

[mheon]

Commands like 'ps' can't be read only - they have to update the DB is they detect a state change in a container. Otherwise we lose things like the exit code and exit time, and think containers that have already stopped are still running

…

On Sun, Nov 8, 2020, 06:28 Daniel J Walsh ***@***.***> wrote: I don't think @yangm97 <https://github.com/yangm97> is after full support, it would just like to run the read/only type functions, podman version, podman info, podman ps. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8253 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCFTKIDJJB3JOCZRLYLSOZ6GFANCNFSM4TL4JGQQ> .

[yangm97]

I don't think we can support r/o - the container state has to be stored somewhere (PID, IP addresses, etc). It may be feasible to store the DB on a tmpfs but we'd lose track of new containers on a reboot.
…

Unless you manage your containers using some supervisor (such as systemd units with the --new flag, which is how I'm doing it right now). Aside from that, you also need to store cni data in a tmpfs if you want to create container interfaces/networks on the fly (I am using --net=host so I didn't notice this earlier).

I've been using the setup described in OP since I opened this, so far so good. I believe it would be cool if this kind of setup was documented. Although it misses persistent volume metadata, that could be fixed by storing the volumes db inside the volumes directory, which has some potential aside from persisting volume ls across reboots (NFS?).

Or maybe podman could fallback to tmpfs like journald and systemd do, possibly frown at the user if they attempt to do some sort of purposeful state modification (to set aside from implicit state modifications such as when you do container ps).

For instance, creating (without --rm, to make explicit this container is intended to be ephemeral)/removing containers, pulling images, etc. would not be allowed but starting existing containers would be fine.

That would be more restrictive than simply moving the graph to tmps as I did, but would still be permissive enough to allow your application to boot despite your rootfs going read-only due some unexpected fs issue.

[rhatdan]

Please open a PR to document this. Or we have just started using the WIKI assocated with this github, you could document this there.

https://github.com/containers/podman/wiki

[yangm97]

Done.