What is happening during the step Deploying container image - it is slow and I would like to to try to make it faster

[cgwalters]

An anonymous user asks: What is happening during the step Deploying container image - it is slow and I would like to to try to make it faster (by throwing hardware at it)

[cgwalters]

That step is copying the input container image to the target - typically a disk image.

Taking the case of e.g. https://github.com/osbuild/bootc-image-builder - it is copying from the "osbuild store" (xref osbuild/bootc-image-builder#466 ) to the target disk.

When using podman run <image> bootc install to-filesystem more directly, it's always copying from the /var/lib/containers storage in the host environment (i.e. the image pulled itself).

Perhaps the easiest way to make this faster externally is to turn off fsync for the target disk image - and the easiest way to do that is to use the volatile option for overlayfs (i.e. make a transient overlayfs with that enabled).

There's a few related things here. One is definitely ostreedev/ostree-rs-ext#365

Another is we could investigate adding some internal parallelism inside the storage writes (in theory, even going to io-uring).

One thing I've been meaning to investigate is chainging the ostree store to use O_DIRECT when doing installs; I suspect this would be a nontrivial speedup alongside doing parallel writes.

But, from an external perspective at the moment the best I can think of is turning off fsync.

[cgwalters]

#655 turns off individual fsync at install time, which definitely helps here.

[brianwcook]

isn't fsync already disabled when running in podman? We are running like this:

time sudo podman run --authfile=$BUILD_DIR/.docker/config.json --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v $(pwd)/config.toml:/config.toml -v $(pwd)/output:/output -v /var/lib/containers/storage:/var/lib/containers/storage $BOOTC_BUILDER_IMAGE $IMAGE_TYPES --local $SOURCE_IMAGE

[cgwalters]

All the writes here are going to the target for $(pwd)/output - what filesystem type is that?

[brianwcook]

good catch! looks like /output is on XFS

[cgwalters]

The general problem here strongly related to how e.g. RUN dnf -y install inside a container image is actually quite needlessly inefficient because it fetches all the RPMs, then unpacks them all one by one, file by file. It also carefully writes things it's doing to the rpm sqlite database which in turn is careful to be "atomic" and fsync things. But since inside a container build, we don't care about any of that - if we get interrupted we can just throw away everything and try again. This is what motivated the addition of the volatile option to overlayfs, which basically ignores the careful fsyncs that sqlite is doing.

But, none of that helps with the parallelism or O_DIRECT side. At least on the bootc/ostree side have Rust now which gives us memory safety without data races, nice high level APIs like rayon, tokio etc. and even high level safe bindings to io-uring. But...the actual writes happening still are in the C library, much like librpm unfortunately.

[cgwalters]

I did some looking at the code to do more things in parallel here (ostreedev/ostree-rs-ext#647 was a trivial prep for this) but...fundamentally our task here is mostly copying a bunch of files plus some metadata, which (if we decided to turn off checksum verification while we do the copying) could be way more efficient if we just basically did a splice/copy_file_range in parallel and didn't even bounce data into userspace.

There's a whole lot of layers of abstraction here that get in the way; also, this strongly relates to https://github.com/containers/storage/issues/1849

[brianwcook]

XFS doesn't have nobarrier option, but I tried echo "write through" | sudo tee /sys/block/nvme0n1/queue/write_cache)

unfortunately it actually made it slower.

[cgwalters]

I did #654 which will help us also output other information such as I/O rates etc.