failed to boot fedora coreos on s390x: fw_cfg option doesn't work

[skycastlelily]

Describe the bug
I try to boot fedora coreos on s390x host following this page[1]:
https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-libvirt/

[root@ibm-z-519 ~]# virt-install --connect="qemu:///system" --name="${VM_NAME}" --vcpus="${VCPUS}" --memory="${RAM_MB}"         --os-variant="fedora-coreos-$STREAM" --import --graphics=none         --disk="size=${DISK_GB},backing_store=${IMAGE}"         --network bridge=virbr0         --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${IGNITION_CONFIG}"

Starting install...
Allocating 'fcos-test-01.qcow2'                                                                                                                                                                                        |    0 B  00:00:00 ... 
Removing disk 'fcos-test-01.qcow2'                                                                                                                                                                                     |    0 B  00:00:00     
ERROR    internal error: qemu unexpectedly closed the monitor: 2022-09-26T08:05:28.680356Z qemu-system-s390x: -fw_cfg name=opt/com.coreos/config,file=/root/test.ign: fw_cfg device not available
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start fcos-test-01
otherwise, please restart your installation.

Which means the qemu-system-s390x emulator doesn't support FW CFG, do we have other methods to pass Ignition file?

[dustymabe]

You have to use a block device. I don't have the virt-install options handy but you should be able to figure them out from the qemu commandline arguments we are using. See:

• providers/qemu: support Ignition block device on s390x and ppc64le  ignition#905 (comment)
• https://github.com/coreos/coreos-assembler/blob/9ee3e7395c0a6889377fa928304754d8961debe9/mantle/platform/qemu.go#L1445-L1450

[skycastlelily]

Thanks a lot for your information:)
The qemu commandline listed on this page[1] doesn't work for me

[root@ibm-z-514 ~]# qemu-system-s390x  -m 2048 -cpu host -nographic -snapshot -drive if=virtio,file=${IMAGE}  -nic user,model=virtio,hostfwd=tcp::2222-:22 -drive file=${IGNITION_CONFIG},if=none,format=raw,readonly=on,id=ignition -device virtio-blk,serial=ignition,drive=ignition
qemu-system-s390x: CPU definition requires KVM

After I remove the -cpu option,it still fails:

[root@ibm-z-514 ~]# qemu-system-s390x  -m 2048  -nographic -snapshot -drive if=virtio,file=${IMAGE}  -nic user,model=virtio,hostfwd=tcp::2222-:22 -drive file=${IGNITION_CONFIG},if=none,format=raw,readonly=on,id=ignition -device virtio-blk,serial=ignition,drive=ignition
LOADPARM=[        ]
Using virtio-blk.
Using guessed DASD geometry.
Using ECKD scheme (block size  4096), 
! Cannot read block 0 on DASD !

[1]https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-qemu/

[skycastlelily]

Tried the following command just now, turns out it works well,so my problem is solved,thanks for your information again.
Just want to learn why the above qemu command line failed.

[root@ibm-z-514 ~]#virt-install --connect="qemu:///system" --name="${VM_NAME}" --vcpus="${VCPUS}" --memory="${RAM_MB}"         --os-variant="fedora-coreos-$STREAM" --import --graphics=none         --disk="size=${DISK_GB},backing_store=${IMAGE}"         --network bridge=virbr0 --qemu-commandline="-drive file=${IGNITION_CONFIG},if=none,format=raw,readonly=on,id=ignition -device virtio-blk,serial=ignition,drive=ignition,devno=fe.0.0007"
...
Fedora CoreOS 36.20220906.3.2
Kernel 5.19.6-200.fc36.s390x on an s390x (ttysclp0)

SSH host key: xxxx
Ignition: ran on 2022/09/27 10:03:45 UTC (this boot)
Ignition: user-provided config was applied
Ignition: Fetching the Ignition config via the Virtio block driver is currently experimental and subject to change.
Ignition: wrote ssh authorized keys file for user: cloud-user
localhost login: cloud-user
Password: 
Fedora CoreOS 36.20220906.3.2

[dustymabe]

@jebon pointed me at coreos/ignition#928 (comment)

so this should work instead of passing --qemu-commandline= directly:

virt-install ... --disk path=$PWD/config.ign,format=raw,readonly=on,serial=ignition

[dustymabe]

I opened coreos/fedora-coreos-docs#467 to address documentation updates for this.

[dustymabe]

Also worth linking to coreos/ignition#928 here.

[skycastlelily]

so this should work instead of passing --qemu-commandline= directly:
virt-install ... --disk path=$PWD/config.ign,format=raw,readonly=on,serial=ignition

Thanks for your information again,it works well.I tried a similar command yesterday without seeing jebon's comment ^^ but it failed as following,that's the reason why I tried qemu-system-s390x command line,turns out I forgot specify IMAGE param.

[root@ibm-z-514 ~]# virt-install --connect="qemu:///system" --name="${VM_NAME}" --vcpus="${VCPUS}" --memory="${RAM_MB}"         --os-variant="fedora-coreos-$STREAM" --import --graphics=none         --disk="size=${DISK_GB},backing_store=${IMAGE}"         --network bridge=virbr0 --disk path=/var/lib/libvirt/images/test.ign,format=raw,readonly=on,serial=ignition

Starting install...
Allocating 'fcos-test-01-9.qcow2'                                                                                                                                                                                      |    0 B  00:00:00 ... 
Creating domain...                                                                                                                                                                                                     |    0 B  00:00:00     
Running text console command: virsh --connect qemu:///system console fcos-test-01
error: The domain is not running

WARNING  Console command returned failure.
Domain creation completed.
You can restart your domain by running:
  virsh --connect qemu:///system start fcos-test-01