nixos: iso-image: use syslinux bootloader for USB booting support

[bobvanderlinden]

This changes the bootloader for iso generation from Grub to syslinux. In addition this adds USB booting support, so that "dd" can be used to burn the generated ISO to USB thumbdrives
instead of needing applications like UnetBootin.

Most of this is based on the ISO generation methods that is used for ArchLinux.

---

Prebuilt iso by @lethalman, updated 26 Feb 2015, based on unstable release of 25 Feb 2015 (commit a869c83):

• Graphical ISO x86_64-linux: http://lethal.noip.me/dl/nixos-graphical-15.05pre56789.gfedcba-x86_64-linux.iso (sha256: 646cc4e699613cae436cab7c22207603ef0c0995a985b6d76507e79137c4ad0a)
• Minimal ISO x86_64-linux: http://lethal.noip.me/dl/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso (sha256: 97b15d201c37a8c7a19a6b60e5d24bc82dcd83edb42dabfc7a78ed23bdd3517b)

---

Shared doc for tests: https://docs.google.com/spreadsheets/d/103doEIjC_Sk_nkmSq9stkVD3tbh4st_4JKYLlXAel84/edit#gid=1631478778

[lucabrunox]

Closes #2100

[lucabrunox]

Add the /g flag to sed?

[bobvanderlinden]

To test this, I've used the following:

$ nix-build nixos/release.nix -A iso_minimal.x86_64-linux

# CD-rom with legacy bios
$ qemu-system-x86_64 -cdrom $(cat result/nix-support/hydra-build-products | cut -f3 -d' ')

# CD-rom with UEFI bios
$ qemu-system-x86_64 -bios /usr/share/ovmf/x86_64/ovmf.bin -cdrom $(cat result/nix-support/hydra-build-products | cut -f3 -d' ')

# USB with legacy bios
$ sudo qemu-system-x86_64 -device piix3-usb-uhci -drive id=my_usb_disk,file=$(cat result/nix-support/hydra-build-products | cut -f3 -d' '),if=none -device usb-storage,drive=my_usb_disk

# USB with UEFI bios
$ sudo qemu-system-x86_64 -bios /usr/share/ovmf/x86_64/ovmf.bin -device piix3-usb-uhci -drive id=my_usb_disk,file=$(cat result/nix-support/hydra-build-products | cut -f3 -d' '),if=none -device usb-storage,drive=my_usb_disk

As for real hardware:

I've tested this on an Asus 1215N. It has a legacy bios and it booted the ISO from USB as well as CD-rom.
I've also tested this on a Macbook Air. It has a UEFI bios and it booted the ISO from USB as well as CD-rom. The USB/CD-rom were emulated by a Google Nexus 5 through DriveDroid.

[MP2E]

I'm in full support of this, the generated ISO from my machine works as intended.

[bobvanderlinden]

@MP2E Awesome, thanks for testing! What did you use to test the ISO? Virtual or physical system? CD/USB? UEFI or legacy bios? I'm fairly certain all combinations should work, but it would be nice to have all combinations checked one way or another.

[MP2E]

I tested it with legacy BIOS on my real machine via a random 8GB USB drive I had lying around

[edolstra]

Note that we actually used isolinux in the past, but we replaced it by GRUB to have a unified boot configuration. For instance, we can have memtest86 both on the ISO and on installed systems using the same memtest86 module, rather than needing separate memtest86 support for the ISO.

Maybe we can introduce some abstraction (say boot.entries) that could be implemented by boot loader modules like grub and syslinux and that could be used by modules like memtest86.

[lucabrunox]

@edolstra I believe having a working iso installer outweighs the need to have common boot entries, since it happens (also happened to me) that unetbootin iso does not work at all.
If a memtest is wanted in the iso, just put it there always regardless of the host configuration.
I agree with you, but I'd please you not to make that feature a blocker for 14.11. Further features can be implemented afterwards.

[domenkozar]

+1 for easier USB booting

[bobvanderlinden]

@MP2E Sounds good. I guess it's also the most used setup for the ISO.
@edolstra @lethalman boot.entries does sound nice. Same goes for Gummiboot, which is now also handled separately.
I do agree with @lethalman that releasing a version soonish is better than waiting for the abstraction. Most people won't be generating ISOs with their own boot entries anyway (other distros have no support for this at all). While working on this I've seen two instances where people had trouble with UnetBootin on IRC, so a fix soon is highly preferable I think.

[bobvanderlinden]

I enabled bootable, efiBootable and usbBootable by default. It might be better to just remove these options and/or move the functionality to something like make-iso9660-syslinux-image.nix instead of make-iso9660-image.nix.

Edit: hmm, I had mistaken that the assertions (https://github.com/bobvanderlinden/nixpkgs/blob/e3409ae541c1f9f346224052d920c61b278c09a7/nixos/lib/make-iso9660-image.nix#L45) are in place. Bad idea to enable these by default (build fails).

[lucabrunox]

In the nixos release we should enable makeUsbBootable, otherwise the nixos installer in the homepage won't be usb bootable. The file is installer/cd-dvd/installation-cd-base.nix, there makeUsbBootable must be true.

[bobvanderlinden]

The same goes for EFI bootable. Should EFI also be enabled by default? Most modern motherboards and laptops are using UEFI to boot nowadays, so this is also most preferable I'd presume.

[lucabrunox]

It's already enabled for the installer. It's that you added the usb, and now needs to be enabled for the installer.

[bobvanderlinden]

Ah good point. I've added the option to installation-cd-base.nix

[lucabrunox]

This is ready. @edolstra do you think the shared boot entries are a blocker? Can you point to a use case where that's necessary currently?

[lucabrunox]

@bobvanderlinden last thing, did you actually need that /g for the sed? That thing is broken anyway, so I'd rather leave that change out.

[bobvanderlinden]

@lethalman Agreed, I've removed it. In the off-chance that there are files with '=' in them, the ISO generation will fail. I think this is the proper behavior until proper escaping is in place.

[lucabrunox]

I've left the old sed code. The escaping must be done when adding files, not after with sed. Merged thanks!!!

[FrozenCow]

Awesome, thanks!

[lucabrunox]

Wrote something in the wiki here, feel free to improve: https://nixos.org/wiki/Installing_NixOS_from_a_USB_stick

[edolstra]

Does Unetbootin still work with this?

@lethalman I'd prefer if you had waited for an explicit +1 for a major change like this, since in a sense this is a regression to the bad old situation of having to maintain multiple boot loader infrastructures...

[lucabrunox]

@edolstra it's sad you want to revert, I think isolinux offers more for booting live cd than grub and that's why most major distros use it. I can revert and we'll rework on it after 14.11

[edolstra]

There seems to be a problem with the VM tests:

$ nix-build nixos/release.nix -A tests.installer.lvm.x86_64-linux
...
building /nix/store/p1j3aks20rr41xmfq05qxdn4zilfw9m4-vm-test-run-installer-lvm
running the VM test script
Formatting 'harddisk', fmt=raw size=4294967296 
machine: starting vm
machine# qemu-kvm: boot=on|off is deprecated and will be ignored. Future versions will reject this parameter. Please update your scripts.
machine: QEMU running (pid 21575)
machine: must succeed: echo hello
machine: waiting for the VM to finish booting
machine# Warning: vlan 0 is not connected to host network
machine# 
machine# ISOLINUX 6.03   Copyright (C) 1994-2014 H. Peter Anvin et al
error: timed out waiting for the VM to connect
timed out waiting for the VM to connect
cleaning up
killing machine (pid 21575)
/nix/store/p1j3aks20rr41xmfq05qxdn4zilfw9m4-vm-test-run-installer-lvm

Is isolinux configured to auto-boot after a few seconds?

[vcunat]

Tested unetbootin on a few computers, some with EFI and some without. NixOS boots fine (memtest boots NixOS, which isn't changed by this PR, as noted above). Now there's nothing left to be tested/improved in this PR and it should get merged to master, agreed?

[bobvanderlinden]

@chisheel @vcunat Thanks for testing. Yes, I think all problematic cases have been covered now with the latest ISO.

I'm all for getting this merged well before the next milestone is released (15.05). That way there is a bigger chance people come across potential problems. If there are problems I'm guessing it is in incompatiblities in options. I think those are covered, but it's hard to be sure without people trying it with their own configurations. So, yes, merging would be a good option at this time I think.

[lucabrunox]

I think it can be merged, @edolstra ?

On Sat, Mar 7, 2015 at 4:19 PM, bobvanderlinden notifications@github.com
wrote:

@chisheel https://github.com/chisheel @vcunat
https://github.com/vcunat Thanks for testing. Yes, I think all
problematic cases have been covered now with the latest ISO.

I'm all for getting this merged well before the next milestone is released
(15.05). That way there is a bigger chance people come across potential
problems. If there are problems I'm guessing it is in incompatiblities in
options. I think those are covered, but it's hard to be sure without people
trying it with their own configurations. So, yes, merging would be a good
option at this time I think.

—
Reply to this email directly or view it on GitHub
#4678 (comment).

NixOS Linux http://nixos.org

[lucabrunox]

Btw let's not merge this before hydra is fixed, so we can test with latest unstable changes.

[vcunat]

BTW, we might better increase this limit, as some people tend to get disk-full here.

[bobvanderlinden]

What limit would you suggest? 20M? 50M?

Or should we calculate what is needed to fit ${efiDir} in there?

[vcunat]

Anyway, it's a thing unrelated to merging this PR.

I would personally prefer to be adaptive, as various per-user config changes may affect the needed size (e.g. kernel modules). Perhaps run du on the things we want there and add several percent for overhead. (We could even re-iterate on each failure with several percent added.) I will produce some code and file a separate PR for anyone interested to review.

[vcunat]

See bobvanderlinden@e478354cd4effa#commitcomment-10190892.

[kylescottmcgill]

If i may, increasing this to 100MB or 512MB would be ideal, on install the EFI Parition needs to be 512MB to allow for multiple initrd's etc and also its more the defacto see: http://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface#Linux

Will also allow people to put other images on to the USB stick should people want to dual boot it. Just an opinion.

[bobvanderlinden]

Yes, but that would mean all ISO images will increase in size quite a bit. Ideal would be to measure how much is needed and create an image that fits those needs. Not sure if du is reliable for that (now much does FAT metadata use?), but it's a good step in the right direction I think.

Also, I've created #6796, so that we have a better place for this conversation. (I have trouble finding these in-code conversations each time ;)).

[bobvanderlinden]

It seems the evaluation errors on hydra are resolved: http://hydra.nixos.org/jobset/nixos/trunk-combined#tabs-errors

Is everything ready for a merge?

[lucabrunox]

👍 for me because installer tests passed the last few days, e.g. http://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.installer.simple.x86_64-linux

[vcunat]

Unfortunately, atop current master the dd-method no longer works for me: it claims there's no bootable media. (I tested with cleaned history of the PR https://github.com/vcunat/nixpkgs/commits/v/iso-full, but I verified the tree is the same as with merge.)

[vcunat]

I'm getting a brief flash of "isolinux.bin is missing or corrupt". I see the file "/isolinux/isolinux.bin" on the media; it has several bytes on the start different from ${syslinux}/share/syslinux/isolinux.bin. I tried two drives already, both with 64-bit minimal image.

[bobvanderlinden]

@vcunat Since you mention isolinux.bin I presume this was on legacy bios hardware? At the moment I only have UEFI hardware available, but I can't reproduce this with qemu. I used commit e19ac24 like you have in v/iso-full.

I also see differences in the first bytes of isolinux.bin on the ISO compared to the original file. I don't know the details, but it could be intentional. It could have something to do with eltorito-boot option, though I'm not sure at this time. I also do not know whether this is indeed related to the problem you're seeing?

EDIT: I removed all result directories before generating the image. The directory is clean. The image that was generated was named:

/nix/store/6b50r6vfqnjiwcbjwc30lpx1i5wszi9s-iso9660-image/iso/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso

I tried it with qemu using:

cp /nix/store/6b50r6vfqnjiwcbjwc30lpx1i5wszi9s-iso9660-image/iso/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso /tmp/
sudo qemu-system-x86_64 -device piix3-usb-uhci -drive id=my_usb_disk,file=/tmp/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso,if=none -device usb-storage,drive=my_usb_disk

(qemu wants a writable ISO file if it emulates USB, that's why I first copied it)

[vcunat]

I tried the two USB disks on an UEFI system now, and it failed (just black screen). Note that one of the images linked above did work on that machine before, just as it did on my main notebook where it fails now. (I'm sorry for my slow latency these weeks.)

EDIT: after the first failures, I was testing the parent of the commit you did 1d649edf2b to rule out an error in the last one (I didn't find that likely and now that commit was picked to master).

[bobvanderlinden]

It seems that I get different results from the build process than @vcunat or @lethalman. I presume it has something to do with useChroot. I have tried building the image on 2 machines, one ArchLinux with chroot disabled, one on NixOS with nix.useChroot enabled in configuration.nix. Both are booting through Qemu and I also tested the image generated with NixOS on my UEFI hardware. There must still be something different in my environment that makes my builds result in different images. Just to be sure, how can I verify whether useChroot is actually working on NixOS?

@vcunat
Just to be sure it isn't hardware: does booting through Qemu for you at all?

Presuming it won't boot through Qemu. Is there a way to verify that we are indeed building the exact same system with exactly the same environment? What path does your build result in? Is it also /nix/store/6b50r6vfqnjiwcbjwc30lpx1i5wszi9s-iso9660-image/iso/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso?

[vcunat]

OK, so now I created clean clone, then removed the .git directory as it seems to affect the derivation names (!), and checked for nonexistence of ./result. I used the same commit v/iso-full e19ac24 to have the determinism changes.

$ nix-build nixos/release.nix -A iso_minimal.x86_64-linux
/nix/store/4dmnrgy3fpvwicxrv60x6plzvdvk6asv-nixos-iso-15.05pre56789.gfedcba

which links to

$ sha256sum /nix/store/0agy1vszv3ifdrzvff8gdmfn557n2dg9-iso9660-image/iso/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso
2a3dee41aac9403cc46b4102468032d20d3a90355eee90214a2ee045747849b4  /nix/store/0agy1vszv3ifdrzvff8gdmfn557n2dg9-iso9660-image/iso/nixos-minimal-15.05pre56789.gfedcba-x86_64-linux.iso

@bobvanderlinden (or others): can you reproduce this? (if you remove .git... I tested that touching files shouldn't affect the derivation names at least, and probably not even the contents)

[vcunat]

Oh, that iso does boot for me with dd + both UEFI and non-UEFI. I have no idea what was wrong before, probably some random error or a mistake of mine.

[vcunat]

Re-tested also unetbootin + UEFI. I think we can merge the branch.

[vcunat]

In master now! @bobvanderlinden: great many thanks for you half-year effort.

[lucabrunox]

Thanks a lot @bobvanderlinden that was a great contribution. Hopefully I don't have to revert this again :P

[bobvanderlinden]

🎉 Yay! Thanks for the merge! @lethalman Yes, let's hope so ;)

I'm still interested why @vcunat had different results earlier. @vcunat Are you sure it was an error on your side and not something else? It seems in the course of this PR there have been instances where I didn't know how to reproduce an error because of the image that was built.

Also (maybe related), do we consider having .git (or even 'result') in the repo non-deterministic? It's only for ISO generation (right?), but it still doesn't seem right.

[vcunat]

I could try to find the old ISO on the other computer (not near me now), but finding what exactly differs etc. is likely to be quite complex.

I think this non-determinism all comes from the fact that the source directory is copied into the ISO as the default channel, and the copy includes everything in the directory. We should at least remove .git.

[vcunat]

Related: any idea on #7132 (comment)?

[lucabrunox]

I don't think its' related in any way to this PR. It's known that sometimes the installer boot fails to find the iso image for mounting. The random fails people see is because something went randomly wrong at burning, or at booting. Nothing related to syslinux or grub.