Warning: /dev/disk/by-label/root does not exist

[jonathanfemideer]

I am using Heads v0.1.0 on a Lenovo X230.

As described here, I am trying to install Qubes on Heads. I am skipping, for now, configuring the PGP key and the TPM, and going straight to the "Installing Qubes" section of Installing Heads. Those steps aside, and with a few additional steps added (see below) I have proceeded as far as this section of those instructions:

TODO: write start-xen script, figure out where the root UUID comes from.
Run /boot/start-xen and kexec -e and wait for the Qubes configuration to finish. The defaults are fine.

To be clear, here are the steps I have followed so far:

• build Heads on a spare PC, yielding x230.rom with SHA256 hash 83df0b6845fdd6b335119f7e89e08c47ec3566f515fe93bc528fcd955587a43e;
• use a Bus Pirate to dump the X230's 4MB and 8MB flash chips' contents into backup files on the spare PC;
• use a Bus Pirate to flash x230.rom to the 4MB flash chip in the X230;
• run me_cleaner on the X230's 8MB chip dump;
• flash the cleaned 8MB rom to the 8MB flash chip in the X230;
• run make xen.intermediate to generate build/xen-4.6.3/xen/xen.gz as described here and here;
• prepare a Qubes R3.2 USB drive as an installation medium, per the official instructions;
• add a third partition to the USB drive, as suggested here and here;
• copy build/xen-4.6.3/xen/xen.gz to that partition;
• boot into Heads on the X230, and insert the USB drive;
• run:

mkdir /tmp/part3
mount -o ro /dev/sdb3 /tmp/part3
cp /tmp/part3/xen.gz /
gunzip /xen.gz
umount /tmp/part3

mount -o ro /dev/sdb2 /boot
cd /boot/efi/boot
kexec -l \
  --module "./vmlinuz inst.stage2=hd:LABEL=Qubes-R3.2-x86_64 i915.preliminary_hw_support=1" \
  --module "./initrd.img" \
  --command-line "no-real-mode reboot=no" \
  /xen
kexec -e

• go through the Qubes OS graphical installer, to install Qubes R3.2 onto an SSD in the X230 (using the default options for now, instead of the "1G for /boot ... 32G for /, 32G for swap and the rest for /home" partition scheme suggested here);
• reboot;
• run:

mkdir /tmp/part3
mount -o ro /dev/sdb3 /tmp/part3
mount -o rw /dev/sda1 /boot
cp /tmp/part3/xen.gz /boot/
gunzip /boot/xen.gz
umount /tmp/part3
umount /boot

• use vi to edit /bin/start-xen as follows:

#!/bin/sh
mount -o ro -t ext4 /dev/sda1 /boot

die() { echo >&2 "$*"; exit 1; }

XEN=/boot/xen
INITRD=/boot/initramfs-4.4.14-11.pvops.qubes.x86_64.img
KERNEL=/boot/vmlinuz-4.4.14-11.pvops.qubes.x86_64

echo "+++ Checking $XEN"
# gpgv "${XEN}.asc" "${XEN}" || die "Xen signature failed"

echo "+++ Checking $INITRD"
# gpgv "${INITRD}.asc" "${INITRD}" || die "Initrd signature failed"

echo "+++ Checking $KERNEL"
# gpgv "${KERNEL}.asc" "${KERNEL}" || die "Kernel signature failed"

# should also check xen command line arguments!
# should also check kernel command line arguments!

kexec \
	-l \
	--module "${KERNEL} root=LABEL=root rhgb" \
	--module "${INITRD}" \
	--command-line "no-real-mode reboot=no console=vga dom0_mem=min:1024M dom0_mem=max:4096M" \
	"${XEN}"


echo "Ready to start Xen: run 'kexec -e' to execute it"

• run /bin/start-xen;
• run kexec -e.

Everything went pretty smoothly until a few minutes after that last step. It produced the usual wall of text that precedes the Qubes graphical installer, and then produced a screen showing the Qubes logo, which asked for my disk encryption password. I provided it, and got a screen with the Qubes logo again, and a slowly-advancing progress bar. But after a few minutes, this suddenly disappeared and was replaced with a text terminal. Here is the tail of that terminal's output:

[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
            Starting Dracut Emergency Shell...
[  199.541761] audit: type=1131 audit(1487996842.208.14): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=plymouth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal? res=success'
Warning: /dev/disk/by-label/root does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


dracut:/#

Perhaps this is a Qubes issue rather than a Heads issue. I'm not sure yet. Will update if discover more. Feedback welcome.

[jonathanfemideer]

At that prompt:

dracut:/# ls /dev/disk/
by-id  by-uuid
dracut:/# ls /dev/disk/by-label
ls: cannot access /dev/disk/by-label: No such file or directory

[jonathanfemideer]

So, I guess the problem is related to the line

--module "${KERNEL} root=LABEL=root rhgb" \

in /bin/start-xen. For that line to succeed, I guess the Qubes kernel would have to be able to find /dev/disk/by-label/root, but it can't, so it borks.

Presumably this explains the "TODO: ... figure out where the root UUID comes from." remark here. With the UUID, we would not have to rely on a label that might turn out not be visible.

[jonathanfemideer]

I wondered if the issue above might have been caused by corruption when dd-ing the Qubes R3.2 ISO to my USB drive. The Qubes installer has a "Verify" button, for verifying the installation medium. So, I booted back into the Qubes installer from the USB drive so that I could hit that button. Sure enough, after a scan, it reported that the drive was not good to install from.

[jonathanfemideer]

I wondered if the issue above might have been caused by corruption when dd-ing the Qubes R3.2 ISO to my USB drive. The Qubes installer has a "Verify" button, for verifying the installation medium. So, I booted back into the Qubes installer from the USB drive so that I could hit that button. Sure enough, after a scan, it reported that the drive was not good to install from.

On closer investigation, that report might have just been due to the presence, on the USB drive, of the extra partition that I created in order to transfer the xen.gz file to the X230.

I re-verified the signature on the Qubes R3.2 ISO, re-dd'd the ISO to the USB drive, booted the Qubes installer afresh, and pressed the "Verify" button again. The installer reported the medium was good to install from.

I installed Qubes again, from the USB drive to the X230's SSD, and upon rebooting, hit the same problem as before:

Warning: /dev/disk/by-label/root does not exist

This, and the fact that I have installed Qubes from this ISO onto other machines with normal BIOSes without issue, makes me think it really is a Heads problem that we're faced with here.

[jonathanfemideer]

Next, I used the blkid command at that prompt in order to find the UUID of /dev/mapper/qubes_dom0-root, which I assume is what the root= parameter from /bin/start-xen was intended to end up pointing at.

Then I tried booting Qubes from Heads again. This time, I changed the following line in /bin/start-xen from:

	--module "${KERNEL} root=LABEL=root rhgb" \

to:

	--module "${KERNEL} root=UUID='XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX' rhgb" \

where XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX is the UUID I found from running blkid above. (I don't want to report the real UUID here in public.)

However, this, too, failed in the same way as before, only instead of:

Warning: /dev/disk/by-label/root does not exist

this time, the warning was:

Warning: /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX does not exist

I don't understand why this would happen.

The resulting console doesn't give me an answer. From what I can see, /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX does exist:

dracut:/# ls /dev/disk/
by-id  by-uuid
dracut:/# ls -l /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
lrwxrwxrwx 1 root 0 10 Feb 25 09:41 /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX -> ../../dm-2

As things stand, I'm stuck.

[rofl0r]

does the symlink target of /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX exist ?

[osresearch]

Your disk image is probably fine -- that doesn't look like a corruption issue.

When I installed Qubes I set the labels on the partitions; this is not listed in the install steps but should be. Otherwise the install needs to update the script.

I'm surprised that you're getting an error about /dev/disk/by-label/root after you changed the command line in start-xen. That should have fixed it and allowed the dom0 kernel to find its root filesystem. Are you certain that you ran the edited copy of the script? Can you re-run it with sh -x /bin/start-xen to be sure that the right UUID is passed in?

[jonathanfemideer]

@rofl0r:

does the symlink target of /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX exist ?

Yes:

dracut:/# ls -al /dev/dm-2
brw-rw---- 1 root disk 251, 2 Feb 26 01:26 /dev/dm-2

Aside: I notice that my X230's RTC seems to be a few hours ahead of UTC. I don't know how to correct that from within Heads, and it doesn't seem worth reflashing the original proprietary BIOS/UEFI ROM just to correct the RTC. But my X230 is non-networked for the time being, so if the clock is a few hours off, I don't think it will cause any problems.

[jonathanfemideer]

@osresearch:

When I installed Qubes I set the labels on the partitions; this is not listed in the install steps but should be. Otherwise the install needs to update the script.

Useful to know! I will try that when time allows. Unfortunately, neither e2label nor tune2fs is present in Heads or in Dracut, so I guess I will have to either pull the SSD from the X230 and mount it on a spare PC to relabel its partitions, or I will have to reinstall Qubes and set labels in the manual partitioning options of the Qubes installer.

I'm surprised that you're getting an error about /dev/disk/by-label/root after you changed the command line in start-xen. That should have fixed it and allowed the dom0 kernel to find its root filesystem.

Glad it's not just me who thinks this =D

Are you certain that you ran the edited copy of the script?

Yes. Otherwise, I would have received the other error:

Warning: /dev/disk/by-label/root does not exist

Can you re-run it with sh -x /bin/start-xen to be sure that the right UUID is passed in?

Done. It is passed in as it should be. Very weird that it results in Warning: /dev/disk/by-uuid/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX does not exist. Maybe I used the wrong style of quoting? I am new to kexec.

[jonathanfemideer]

@osresearch:

When I installed Qubes I set the labels on the partitions; this is not listed in the install steps but should be. Otherwise the install needs to update the script.

Useful to know! I will try that when time allows. Unfortunately, neither e2label nor tune2fs is present in Heads or in Dracut, so I guess I will have to either pull the SSD from the X230 and mount it on a spare PC to relabel its partitions, or I will have to reinstall Qubes and set labels in the manual partitioning options of the Qubes installer.

I thought I would try something simpler, just as an experiment. I changed the following line in /bin/start-xen from:

	--module "${KERNEL} root=LABEL=root rhgb" \

to:

	--module "${KERNEL} root=/dev/mapper/qubes_dom0-root rhgb" \

It worked. This avoids having to assign labels during Qubes installation and it avoids having to deal with UUIDs. It might be a good option for this to be the approach Heads adopts for future releases. I have submitted a pull request.

Caveat: Using root=/dev/mapper/qubes_dom0-root might not work if the root partition is not encrypted. Heads should maybe check for that anyway and warn the user if it detects that the user has neglected to encrypt their Qubes DOM0 root partition.

[mfc]

I have the exact same issue. unfortunately your workaround doesn't work for me.

--module "${KERNEL} root=/dev/mapper/qubes_dom0-root rhgb" \

encounters

dracut: FATAL: Don't know how to handle 'root=qubes_dom0-root'
dracut: Refusing to continue

while (notice = instead of -)

--module "${KERNEL} root=/dev/mapper/qubes_dom0=root rhgb" \

encounters the same error we had previously:

Warning: /mapper/qubes_dom0=root does not exist

The hard drive with Qubes installed on it is encrypted.

Tried with no luck:
--module "${KERNEL} root=/dev/mapper/luks-*-*-*-*-*=root rhgb" \
--module "${KERNEL} root=/dev/mapper/luks-*-*-*-*-*-root rhgb" \
--module "${KERNEL} root=/dev/mapper/luks-*-*-*-*-* rhgb" \

[JohnnyLeone]

The problem will be that you're partition don't provide a label like qubes_dom0-root. That means you have to specify exactly the name. A simple luks-*-*-*-*-*-root will not work because after --module you specify a string that cant be abbreviated by kexec.

[jonathanfemideer]

@mfc :

unfortunately your workaround doesn't work for me.

--module "${KERNEL} root=/dev/mapper/qubes_dom0-root rhgb" \

encounters

dracut: FATAL: Don't know how to handle 'root=qubes_dom0-root'
dracut: Refusing to continue

Does this drop you into a dracut shell? If so, what is the output of running blkid there?

while (notice = instead of -)

--module "${KERNEL} root=/dev/mapper/qubes_dom0=root rhgb" \

encounters the same error we had previously:

Warning: /mapper/qubes_dom0=root does not exist

That is to be expected, if there is no name qubes_dom0=root in the mapper (for example, if luksOpen has not placed that name into the mapper).

The hard drive with Qubes installed on it is encrypted.

So was mine.

[mfc]

Does this drop you into a dracut shell? If so, what is the output of running blkid there?

holy shit I'm in! didn't do anything differently but this time dracut didn't complain. amazing. so your workaround does work for me now:

--module "${KERNEL} root=/dev/mapper/qubes_dom0-root rhgb" \

I'ill comment the same in your pull request.

[osresearch]

With LVM in the Heads ROM it is now possible to find the right UUID for root in a consistent manner.