Full Discoverable Partitions Specification compatibility (WIP) #98

rdvdev2 · 2021-09-06T01:15:16Z

The current booster Discoverable Partitions Specification support is pretty basic and doesn't work for every case. This PR aims to fully support the specification, allowing to auto-discover encrypted partitions or mount root read-only, among other features.

WARNING: This PR is really a WIP, and not properly tested. At the moment, this PR is only provided to get feedback while developing the feature(s).

Progress:

Unencrypted root support
Encrypted root support (LUKS)
Scan only the disk containing the used ESP
Honour the GPT flags (read-only mounting and partition skipping)
Dm-verity? (Not familiar with it neither sure if Arch or booster support it)
Documentation?
Tests? (Possibly on the GPT flag reading process)
Works on VM (Not tested yet)
Works on real hardware (Not tested yet)

anatol

The idea of implementing discoverable partitions great. The code looks clean, good work! I left a few comments about this implementation.

It needs more tests and I can help you with it if needed (setting up integration tests is tricky currently).

anatol · 2021-09-07T03:10:35Z

init/main.go

@@ -619,6 +622,15 @@ func boost() error {
 		return err
 	}

+	// Mount efivarfs if running in EFI mode
+	if _, err := os.Stat("/sys/firmware/efi"); errors.Is(err, os.ErrNotExist) {
+		if err := mount("efivarfs", "/sys/firmware/efi/efivars", "efivarfs", unix.MS_NOSUID|unix.MS_NOEXEC|unix.MS_NODEV, ""); err != nil {


I think we need to unmount it as well before switching to the user's root fs

Where should I unmount (function)?

SeemoveMountpointsToHost() function. First it tries to move the mountpoint to host. If it does not work (e.g. new root does not contain the directory) then it unmounts the filesystem from initram fs tree.

The moveMountpointsToHost() should handle /sys/firmware/efi mountpoint as well. I wonder if MS_MOVE and MNT_DETACH handle mountpoints recursively.

I've been reading through the documentation and I think that moving the mountpoint isn't needed at all. It looks like systemd handles all the mountpoints but /run. Take a look at the specification, it may even be the case that the current code on moveMountpointsToHost() is mostly unneeded.

It actually used to be like mentioned in this specification. But it turns out that other init systems like runit expect /dev/ to be mounted at the boot time. See #92 and 4c95058

I am fine to leave this code as-is and wait for integration tests results to see if runit tests pass.

Then I'll try to get unit tests ready today and see if it works.

I ran your PR with VoidLinux and ArchLinux+btrfs tests and they passed.

init/luks.go

init/deviceref.go

init/main.go

rdvdev2 · 2021-09-08T15:12:30Z

It needs more tests and I can help you with it if needed (setting up integration tests is tricky currently).

I don't really know how testing works with this project, so help would be really appreciated.

anatol · 2021-09-08T17:15:47Z

I don't really know how testing works with this project, so help would be really appreciated.

The integration test are located at tests directory. Each test contains from 2 parts:

a bash script that creates a host image (see tests/generate_asset_*.sh)
golang test that runs QEMU with this host image

If you can take care of 1 then I'll add QEMU tests for it. See generate_asset_gpt.sh that creates an image with a GPT and multiple partitions. You can even run this script from you shell with OUTPUT=assets/gpt.img FS_UUID=e5404205-ac6a-4e94-bb3b-14433d0af7d1 FS_LABEL=newpart bash -x -o errexit ./generate_asset_gpt.sh it will generate a valid image at assets/gpt.img. You probably want something similar to initialize an GPT with particular partition UUIDs.

As of formatting a partition with LUKS check generate_asset_luks.sh.

anatol · 2021-09-23T01:35:05Z

Hey I just want to check with you how it is going. Do you need any help with testing?

rdvdev2 · 2021-09-23T20:59:06Z

I'm sorry for the inactivity, last week I started university and I've been really busy since then. Now that I have some free days I'll hopefully be able to continue working on this. I'll let you know if I have any problems with the testing setup.

anatol · 2021-09-23T23:26:57Z

init/main.go

@@ -619,6 +622,15 @@ func boost() error {
 		return err
 	}

+	// Mount efivarfs if running in EFI mode
+	if _, err := os.Stat("/sys/firmware/efi"); errors.Is(err, os.ErrNotExist) {


I believe it should be !errors.Is(err, os.ErrNotExist). if body should be executed if the efi dir not-not exists (i.e. exists).

anatol · 2021-09-23T23:28:18Z

I pulled the latest changes from this PR and rebased on top of wip branch.

anatol · 2021-09-23T23:35:57Z

I tried to ran a test that uses autodiscovery (e.g. go test -v -run TestBooster/Gpt.Autodiscovery) and I see that it requires EFI mode for QEMU now. One question here (and sorry if it is dumb one) - does autodiscovery work without EFI?

I enable QEMU EFI mode with -bios /usr/share/ovmf/x64/OVMF.fd parameters and now my test fails with open /sys/firmware/efi/efivars/LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f: no such file or directory. Is this efi var required for autodiscovery? What if it does not exist at a machine and no root= specified, should booster fail into emergency shell? Or is there a way to recover from this situation?

rdvdev2 · 2021-09-24T00:00:22Z

The EFI var is needed to know which physical disk contains the auto-detected system. I think this is done to avoid loading the kernel from a LiveUSB and the root of a locally installed system (and other probably breaking situations). Knowing this, I'm not sure if it would be a good idea to go ahead and autodetect on all drives, unless there's another way to detect where the kernel was loaded from.

Maybe we could still autodetect if there's only one disk that follows the specification or only one of them contains partitions for the system architecture, but in any other case, I think that falling back to the emergency shell would be the most sensible option. Let me know what you think.

anatol · 2021-09-24T00:09:34Z

init/deviceref.go

-				return &deviceRef{refName, calculateDevName(devPath, p.num)}
+		case refEspUUID:
+			if bytes.Equal(d.data.(UUID), p.uuid) {
+				return d.autodetectRoot(devPath, t)


The autodetectRoot() function does not accept a partition but instead gets whole table. For me it looks like the autodetectRoot()` should be split into 2 parts

calculate rootTypeGuid - move outside of the loop

if bytes.Equal(rootTypeGuid, p.typeGUID) { - should be called inside the loop

udev events might be too noisy. Move it to a separate kernel option `booster.debug.udev` to make the main booster debug logs more readable. Closes anatol#99

…ted)

anatol · 2021-09-24T00:18:56Z

init/deviceref.go

+		if enableAutodetect && efiVarsAvailable {
+			debug("%s= param is not specified. Use GPT partition autodiscovery.", name)
+
+			_, data, err := attributes.ReadEfivarsWithGuid("LoaderDevicePartUUID", *util.StringToGUID("4a67b082-0a4c-41cf-b6c7-440b29bb8c4f"))


where does LoaderDevicePartUUID requirement come from? I checked https://www.freedesktop.org/wiki/Specifications/DiscoverablePartitionsSpec/ and do not see any mention of this efi var.

Ok, found it here the first partition with this type UUID on the disk containing the active EFI ESP is automatically mounted to the root directory /

So basically we need to find a disk that contains the active EFI ESP. Is reading LoaderDevicePartUUID efivar and then iterating all GPT partitions the only way to do it?

anatol · 2021-09-29T01:34:43Z

@rdvdev2 how do you feel if i pick this PR and complete it? I would like to do some refactoring of booster code along the way. I'll keep you as an author of this change.

rdvdev2 · 2021-09-30T13:37:17Z

@rdvdev2 how do you feel if i pick this PR and complete it? I would like to do some refactoring of booster code along the way. I'll keep you as an author of this change.

Go ahead, I think I don't have enough spare time to work on this anymore. Thanks for understanding!

anatol · 2021-10-02T22:58:55Z

@rdvdev2 I reworked your patches, made extra changes to booster codebase and added support for GPT partition attributes. I pushed the changes to wip branch. Please test it when you have a chance.

Per the spec https://systemd.io/DISCOVERABLE_PARTITIONS/ only disks with active ESP need to be checked for discoverable partitions. To find out what is active ESP we need to read efi variables at the boot time. Then we skip all GPT tables that do not have this ESP. This change has been started by Roger Díaz Viñolas as a part of PR #98 and later reworked by Anatol Pomozov.

anatol · 2021-10-07T17:01:10Z

The Discoverable Partitions support has been merged master and becomes part of 0.7 release.

Thank you for your work!

rdvdev2 · 2021-10-10T11:51:24Z

Thank you too for your help (and sorry for not being able to finish this myself)

anatol reviewed Sep 7, 2021

View reviewed changes

init/main.go Outdated Show resolved Hide resolved

anatol reviewed Sep 23, 2021

View reviewed changes

anatol force-pushed the dsp-compat branch from 1b27f21 to 11a8f7f Compare September 23, 2021 23:28

anatol reviewed Sep 24, 2021

View reviewed changes

anatol and others added 3 commits September 23, 2021 17:11

Move udev debug logs under its own kernel option

d4dadfe

udev events might be too noisy. Move it to a separate kernel option `booster.debug.udev` to make the main booster debug logs more readable. Closes anatol#99

Move calculateDevName() into its own function

d8caf32

Full Discoverable Partitions Specification compatibility (WIP & untes…

0ffc23a

…ted)

anatol reviewed Sep 24, 2021

View reviewed changes

anatol force-pushed the dsp-compat branch from 11a8f7f to 0ffc23a Compare September 24, 2021 00:22

anatol closed this Oct 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full Discoverable Partitions Specification compatibility (WIP) #98

Full Discoverable Partitions Specification compatibility (WIP) #98

rdvdev2 commented Sep 6, 2021 •

edited

Loading

anatol left a comment •

edited

Loading

anatol Sep 7, 2021

rdvdev2 Sep 8, 2021

anatol Sep 8, 2021

rdvdev2 Sep 11, 2021

anatol Sep 12, 2021

rdvdev2 Sep 12, 2021

anatol Sep 14, 2021

rdvdev2 commented Sep 8, 2021

anatol commented Sep 8, 2021

anatol commented Sep 23, 2021

rdvdev2 commented Sep 23, 2021

anatol Sep 23, 2021

anatol commented Sep 23, 2021

anatol commented Sep 23, 2021

rdvdev2 commented Sep 24, 2021

anatol Sep 24, 2021

anatol Sep 24, 2021

anatol Sep 24, 2021

anatol Sep 24, 2021

anatol commented Sep 29, 2021

rdvdev2 commented Sep 30, 2021

anatol commented Oct 2, 2021

anatol commented Oct 7, 2021

rdvdev2 commented Oct 10, 2021

Full Discoverable Partitions Specification compatibility (WIP) #98

Full Discoverable Partitions Specification compatibility (WIP) #98

Conversation

rdvdev2 commented Sep 6, 2021 • edited Loading

anatol left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdvdev2 commented Sep 8, 2021

anatol commented Sep 8, 2021

anatol commented Sep 23, 2021

rdvdev2 commented Sep 23, 2021

Choose a reason for hiding this comment

anatol commented Sep 23, 2021

anatol commented Sep 23, 2021

rdvdev2 commented Sep 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anatol commented Sep 29, 2021

rdvdev2 commented Sep 30, 2021

anatol commented Oct 2, 2021

anatol commented Oct 7, 2021

rdvdev2 commented Oct 10, 2021

rdvdev2 commented Sep 6, 2021 •

edited

Loading

anatol left a comment •

edited

Loading