OSBuild without compression yields GRUB failures

[dustymabe]

When we switched OSbuild over to not compressing we started getting failures where images wouldn't boot like:

  Booting `Fedora CoreOS 40.20240207.dev.1 (ostree:0)'

error: ../../grub-core/fs/fshelp.c:257:file
`/boot/ostree/fedora-coreos-f25027bf4dfb2213
7b7c8401ed2396924ba112482afc30d62b7a4ab1310b09db/vmlinuz-6.8.0-0.rc0.20240112gi
t70d201a40823.5.fc40.x86_64' not found.
error: ../../grub-core/loader/i386/pc/linux.c:422:you need to load the kernel
first.
 
Press any key to continue...

[dustymabe]

some more info.. it appears something with the ext4 filesystem when OSBuild runs the qemu-img convert to get from raw to qcow2 is getting corrupted. I converted the image back into a raw image from the qcow2 and mount up the third partition and when I ls on the ostree/ dir I see:

bash-5.2# ls /mnt/ostree/
[ 7220.329345] EXT4-fs error (device loop0p3): ext4_lookup:1855: inode #65538: comm ls: iget: checksum invalid
ls: cannot access '/mnt/ostree/': Bad message

[dustymabe]

ok further update.. I became increasingly suspicious of the corruption happening earlier in the OSBuild steps themselves rather than when copying the file out of the supermin VM.

I decided to just add a sanity check into the org.osbuild.qemu OSBuild stage that immediately ran a qemu-img compare after it created the qcow2 from the raw image. Here's the patch:

diff --git a/stages/org.osbuild.qemu b/stages/org.osbuild.qemu
index 642b5146..54e707d4 100755
--- a/stages/org.osbuild.qemu
+++ b/stages/org.osbuild.qemu
@@ -219,22 +219,34 @@ def main(inputs, output, options):
     if coroutines:
         print(f"qemu-img coroutines: {coroutines}")
         extra_args += ["-m", coroutines]
 
     cmd = [
         "qemu-img", "convert",
         "-O", fmt["type"],
         *extra_args,
         source, target
     ]
 
     subprocess.run(
         cmd, check=True
     )
 
+    # Sanity check that the image is 100%
+    cmd = [
+        "qemu-img", "compare",
+        "-f", "raw",
+        "-F", fmt["type"],
+        source, target
+    ]
+    subprocess.run(
+        cmd, check=True
+    )
+
+
     return 0
 
 
 if __name__ == '__main__':
     args = osbuild.api.arguments()
     r = main(args["inputs"], args["tree"], args["options"])
     sys.exit(r)

and sure enough one of the first runs I went through complained:

⏱  Duration: 0s
Pipeline qemu: dd43158cea505a60ebd511dc09e0fba0247cef4903ebf9c5b2c70e813b54a2cf
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.qemu: dd43158cea505a60ebd511dc09e0fba0247cef4903ebf9c5b2c70e813b54a2cf {
  "filename": "fedora-coreos-39.20240214.dev.0-qemu.x86_64.qcow2.tmp",
  "format": {
    "type": "qcow2",
    "compression": false,
    "compat": "1.1"
  }
}
[   97.999904] audit: type=1400 audit(1707944575.724:72): avc:  denied  { mounton } for  pid=961 comm="mount" path="/srv/cache/osbuild/store/tmp/buildroot-tmp-bzjrq14z/inputs/image/disk.img" dev="vdb1" ino=17809672 scontext=system_u:system_r:ke1
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
Content mismatch at offset 403718656!
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.qemu", line 251, in <module>
    r = main(args["inputs"], args["tree"], args["options"])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/osbuild/bin/org.osbuild.qemu", line 241, in main
    subprocess.run(
  File "/usr/lib64/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['qemu-img', 'compare', '-f', 'raw', '-F', 'qcow2', '/run/osbuild/inputs/image/disk.img', '/run/osbuild/tree/fedora-coreos-39.20240214.dev.0-qemu.x86_64.qcow2.tmp']' returned non-zero exit status 1.

⏱  Duration: 4s

Failed

From the Content mismatch at offset 403718656! line we can see that offset 403718656 is ~ 385 MiB which places it squarely in the /boot/ parition 3.

[dustymabe]

Talked to @jlebon on this. We think (actually more @jlebon, I was just nodding my head to everything he said) that this might be a bug with reflinks on XFS.

The cache qcow2 is an XFS filesystem and when we switch to not using compression for the org.osbuild.qemu stage we think it enables qemu-img convert to use reflinks for some of the data ranges in the image (i.e. to share between the source raw image and newly created qcow2 image).

i.e. this is a theory that could explain why we see issues only after switching off compression. This particular issue reminded @jlebon of #935 which we never fully got to the bottom of.