-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade of crouton bricks Chromebook due to dev_boot_signed_only=1 #765
Comments
crouton should not reset the flag (even if the kernel is signed: it's ok to throw a warning, but it should not reset it): we can easily fix that. I'm not aware of a way to force booting the alternate kernel (have you tried doing a few hard resets with Power+Refresh?). You could try booting off a USB disk with a Chromium OS image, but you probably do not have USB booting enabled (otherwise crouton would not have reset My guess is that you have no way of booting your system except by creating a Chrome OS recovery USB disk, which will erase all the content of the drive... (another alternative would be to open the Chromebook, take out the SSD, and connect it to another computer to recover the content, but, well, NGFF adapters are not very common...) |
I'm really sorry about that. I thought the heuristics I have in place to not set the signed_only bit were sufficient, but apparently they were not. Turns out it's pretty hard to figure out if it's safe to set that bit or not. Sounds like we need to do one (or more) of four things:
|
I don't think Crouton should ever automatically change dev_boot_signed_only=1. Since a crouton user is already in developer mode, they should be well aware that settings like enabling usb boot and disabling OS verification are less safe than than the out of box settings of the Chromebook. However, I realize a lot of folks use ONLY Crouton and ChromeOS and they would benefit from having these set for them. So if the text warnings during install and updates aren't enough, please consider having a flag for installs and updates for users who do not want Crouton to change these settings, as we are using our Chromebooks to test non-signed distros, or need features that won't run with the signed os setting, and usb/sd boot disabled. Thanks. |
I'd argue the opposite; only those who really know what they're doing should run make_dev_ssd.sh, manually unset dev_boot_signed_only, and set dev_boot_usb while they're at it to avoid any chance of dev_boot_signed_only to be set. crouton puts out a warning when it sets dev_boot_signed_only, although it'd be more visible if it were at the end. What features don't work with usb/sd boot enabled? If you're testing various distros, you'd be crazy to not have usb/sd boot enabled. |
@dnschneid Perhaps I worded my last message incorrectly. The last time I made up a bootable SD card with Arch linux on it, I needed to use these commands: crossystem dev_boot_usb=1 dev_boot_signed_only=0 I would generally want those commands to "stick" so that anytime I wanted to boot my chromebook with Arch and the SD card, all I have to do is poweroff, insert the SD card, and press ctrl-U to boot arch from SD, though most of the time, the SD card isn't used, and I'm using Crouton and ChromeOS. However, on occasion, I may also want to be able to powerwash and revert the system to chromebook oob non-developer mode (dev_boot_signed_only=1 and dev_boot_usb=0), if say a family member wants to borrow it for a trip and uses ChromeOS only. |
That's covered by the heuristics today. If you set dev_boot_usb, crouton will never touch dev_boot_signed_only. It also checks if rootfs can be mounted read-write, which I thought would cover the non-external-boot cases but apparently does not (sufficiently). That's the issue here. Maybe we just need an additional check if the kernel partition can be mounted read-write as well? |
OK, then my use cases will work with the current heuristics then. Thanks. I am not aware of all of the current possibilities with read-write of non-signed OS's internally. The last time I was looking at booting arch from internal, it required a screw or wire removal from inside the samsung arm, and I wasn't comfortable doing that mod, although the idea of running arch in the chromebook faster, and without the card hanging out was tempting. It sounds like folks now have os's or 3rd party filesystems booting and running internally without hardware mods on the ARM? |
Just to confirm I understand the problem, is the following statement correct? After the switch to dev mode,
Re kernel verification, we could use If we really want to set it back to |
@drinkcat I believe your first sentence is correct. Your 2nd bullet about problems are that with dev_boot_signed_only set to 1, which the link below says persists as long as you stay in dev mode, is that you can't boot non-google signed OS's like chrubuntu, or arch, externally or internally (?) until you reset dev_boot_signed_only back from 1 to 0. In fact you can't even really check the true state of this bit except before boot. Some of the things below are old, for example, netflix runs in dev mode, so these docs below could be old or device dependent: But if the document is correct in that once dev_boot_signed is set to 1 by the user after entering dev mode, or by letting Crouton do it, then crouton updates shouldn't need to reset it, as it should already be set, unless someone explicitly reset it to 0 for a purpose. Crouton could warn users though, in case they forgot and want the security, so a warning is fine during updates, and as long as an alternate boot is not enabled, perhaps Crouton always setting signed boot to 1 is OK on install, and checking/notifying during updates. https://groups.google.com/a/chromium.org/forum/#!msg/chromium-os-discuss/nMZCsQbq_84/K0Uz9iH77_cJ |
The current heuristics have been in place for more than a year. If you haven't noticed it, then it's because they cover 99% of cases. There have been two bugs filed where the heuristics failed; the first was fixed a year ago with the read-write heuristic, the second is this bug here. I'd rather get the heuristics right or do a full validation (I believe there's a script that wraps vbutil_kernel --verify and compares the output against known keys) than add user prompts for scary things and store responses in random parts of stateful. |
I have to say I totally agree with David that it's a lot better for the heuristics to get it right than to emit a scary warning trusting that the user will get it right. I think if you had 50 GB of data wiped out due to setting dev_boot_signed_only=1, you would probably agree :-. (I installed a 128 GB SSD). |
@wchao I'm sorry about your data loss. I've had a few myself, and currently put precious data in ~/Downloads accessible from ChromeOS in case my chroot gets corrupted in any way. I initially liked your suggestion in your 2nd paragraph of your original post:
but then David explained that the current heuristics would not set that if the user has already set dev_boot_usb on (my use case for not wanting crouton to set to 1) so if David's idea above of doing a 2nd check:
would work to prevent the situation you had, and all current known cases of setting dev_boot_signed_only to 1 then that seems like a good approach. Can you confirm that David's idea of the "additional check if the kernel partition can be mounted rread-write" would prevent the issue you just experienced? |
It's not a huge loss (mostly just time). I can recreate all the data (just takes some time sitting there waiting for the machine to install software). Regarding your questions about when to set dev_boot_signed_only to 1, I would say there are several motivations for caring about that flag. First, there is the motivation of added security, which I think we would all agree is good, except when it conflicts with other motivations. A second motivation (in this case, in the opposing direction) is not wanting a bricked machine. I am not sure what relation the kernel partition being read-write capable has to being able to boot the machine (i.e. not having a bricked machine). Suppose the kernel partition can be mounted read-write. Does that imply that the Chromebook will boot a partition that is not signed by Google? If a read-write capable partition always implies that Chromebook will boot that partition, then I would say that is a good heuristic that will ensure a machine never gets bricked (at least from setting dev_boot_signed_only=1). I was not aware that crouton will avoid changing dev_boot_signed_only when it sees dev_boot_usb. Now that I know that, I am definitely going to set dev_boot_usb to 1 when I reformat and reinstall! I also learned that I can set the boot priority using cgpt to try different partitions in order (http://dev.chromium.org/chromium-os/developer-information-for-chrome-os-devices/cr-48-chrome-notebook-developer-information/how-to-boot-ubuntu-on-a-cr-48#TOC-Set-boot-priority). I am going to do some testing with that to see if I can make my configuration more resilient in the future (in case I run into a similar situation). For other unsuspecting people in the future, it would be good to have a foolproof (or as foolproof as practically possible) heuristic for determining when it is safe to set dev_boot_signed_only=1. In my mind, that is checking whether the boot image is signed by Google or not. Is there some way to verify the signature of the boot image using vbutil_kernel or some other tool, and compare to Google's public key? If you could do that, then you could avoid setting dev_boot_signed_only when you know that the next reboot will brick the machine. |
I managed to recover the data. I had another Chromebook and realized that if I took the SSD from the other Chromebook, I would be able to boot. Then I set dev_boot_signed_only=0, and installed the original SSD. Problem solved. |
@wchao that is genius! I wonder why I did not think of that (other than my samsung arm has a soldered in emmc I think). I'm still not sure of the best overall solution here. I know that I personally would want dev_boot_signed_only=0 to stay that way if I set it that way for a purpose, but I can also see that for many Crouton users using the standard unmodified targets and default kernel boot options, that it offers some protection by setting back to 1. |
@wchao: Good job getting your data back ,-) I came up with the following script: https://gist.github.com/drinkcat/10843992 , making use of The logic goes like this:
I only tested signed Chromebooks at this stage (QEMU is a bit irrelevant as there is no flag to start with), @wchao maybe you can help to confirm this works. @DennisLfromGA, since you have Chrubuntu installed, can you check what is the output is your case? Finally, the test takes about 15 seconds (reading the firmware is slow), should we consider caching the result somewhere in |
@drinkcat: I tested your testsig.sh and it works on my system. Here is what I get: Saving verbose log as /tmp/debug_vboot_2sT8b92S6/noisy.log It doesn't take 15 seconds on my Acer C720. It takes about 5 seconds. |
@drinkcat: I also tested when one of the kernel partitions is signed, and it also seems to work: Saving verbose log as /tmp/debug_vboot_RnwdCU0yH/noisy.log What do you think about the following heuristic? If all kernels are signed, then it's safe to set dev_boot_signed_only=1. If all kernels are NOT signed, set dev_boot_signed_only=0. If some kernels are signed and some kernels are NOT signed, then display a warning along with an explanation to the user of what might happen, then give the user the choice (which should default to dev_boot_signed_only=0). In cases where some kernels are signed and some are not, it might work most of the time to default dev_boot_signed_only based on whether the highest priority kernel partition is signed or not, but still would be good to display the warning and get user input because there's no guarantee that the highest priority kernel partition will be able to boot OK. |
@wchao : cool, thanks for trying it out. It does take 15 secs on my C720 (and about the same on my Samsung ARM), weird... I'd say the logic should be:
As @dnschneid mentionned, we should also make sure that these messages gets displayed again in the |
That sounds good, although I would get rid of the RW rootfs check since it's a bunch of code that only avoids the full verification check in relatively few cases. I think the way to go about this is to kick off the verification in parallel at the point where we do the crossystem edits now, then check the result and apply the edits after bootstrap/extraction but before prepare.sh. We can put a small status output when setting the flag, then output the warning message as part of the trap handler so that it outputs at the very end, after prepare.sh has run, regardless of whether prepare.sh suceeded or not. That has the following benefits:
|
Indeed. I only realised now that remounting the rootfs RW requires a kernel command line modification, so it's exactly the same scenario: the kernel(s) will not verify. Now, I like the background idea, but I'm not sure if it's worth it, given how "interesting" multi-tasking can get in shell:
And I agree with points 3 and 4. |
Since we can test the exit code via Caching the result for the few people who do strange configurations and repeatedly install/update chroots per boot is even less reasonable :) |
FYI, to avoid the data loss issue (maybe), as you likely have a Google account with a google drive, Insync runs very well under crouton. At least, under crouton/Xubuntu which i'm running on my C720P. My data is backed up to my google drive, and synced to three other computers at the same time (two linux boxes also running Insync, and one Windows box running Win7 and Google's drive app). |
@drinkcat, et all, Sorry guys, I've been away for a couple of weeks and just got back late yesterday. System:
First of all, I have removed rootfs verification on my system so I can run some custom init scripts and such so this may the sully the results of the script. My dev_boot flags are:
I ran the script and got this output:
Hope this helps in any way. -DennisL |
@DennisLfromGA : Thanks, that confirms what we thought. I still need to think a bit more about the heuristic of @dnschneid : Alright, I guess you convinced me ,-) |
Please bear with me - noob question(s):
Curious, |
wchao commented on Apr 16, 2014 I managed to recover the data. I had another Chromebook and realized that if I took the SSD from the other Chromebook, I would be able to boot. Then I set dev_boot_signed_only=0, and installed the original SSD. Problem solved. I am looking for some clarification on exactly what wchao did and why it worked. I assume the drive from the other Chromebook must have also been in Developer Mode (or the Developer Mode status is stored in the r/w firmware) and had the kernel boot bits stored on the drive were signed by Google, else the dev_boot_signed_only=0 command could not have been executed? Also, The dev_boot_signed_only status must be stored in the r/w firmware area of Chromebooks, and NOT anywhere on the hard drive? Thanks Much In Advance for any clarification! Regards, Ron PS - I had to perform a Recovery from USB after performing a crouton update. After the update and execution of the "sudo startxfce" command, it promptly crashed and rebooted thereafter. The dev_boot_signed_only must of been set to 1 by the update. |
The other Chromebook was in Developer mode. Anecdotally, I seem to recall that in the past few months, crouton seems to again set dev_boot_signed_only=1. It didn't brick my Chromebook because I was careful to check for it after my previous near-trauma, but I'm sure some people are caught up by it. I haven't played with it much recently, so I don't know if it's still doing that as of the last few weeks. |
@rrolsbe : Sorry about that... There is a fix in #785, but it has not been merged yet: we had so few reports that we deviced to leave it aside. We will revisit it at some point... In the mean time, unless you have strong security concerns, I'd suggest enabling USB booting ( |
Thanks for the reply! Since I own the two C710 Chromebooks, (the ones without the Seabios payload, thus not allowing traditional Legacy OS USB booting) what can I boot via USB to access the internal hard/flash drive content? Informational link would be greatly appreciated. Thanks Again!! I think the link below might of worked on the original C710 to allow booting from USB without changing the factory Coreboot firmware. I tried to install this onto three different USB sticks and got the same errors all three times. 1st error - chronos does NOT have the parted GPT partition manager binary. Has anyone on this forum EVER gotten this script to properly install a bootable version of ChrUbuntu on a USB stick? If so, did it boot on the C710 WITHOUT changing the factory delivered firmware? http://chromeos-cr48.blogspot.com/2012/12/so-you-want-chrubuntu-on-external-drive.html |
If you have or can build ChrUbuntu on an external usb stick and On Wed, Jan 14, 2015 at 3:46 PM, rrolsbe [email protected] wrote:
DennyL@GMail |
I noticed an interesting tidbit in the verified dev mode wiki page:
To those potentially affected by this bug (you know who you are, and @divx118 for good measure): could you run |
Actually, the output of |
I am running Chrome OS 42.0.2311.87 (64 bit) with a patched kernel partition (to add disablevmx=off lsm.module_locking=0 to the kernel command line). I get the following crossystem values: crossystem kernkey_vfy: hash |
Sweet. One more thing: next time you (or anyone) get an update such that you have to re-patch your kernel, could you run both commands 3 times: before patching it, after patching it, and after rebooting? |
Sure, happy to. I'm using the latest Chrome OS, so I don't know when the next time will be, but I would guess in 1 - 2 months it's likely. |
If enough people say If it DOES immediately change, then it may also be worth it to add checks to crouton enter_chroot to see if the status changed, and then alert/disable verified boot to prevent people from nuking themselves even outside the context of an install or update. |
@dnschneid If there is an update pending, the next reboot will boot another kernel. My script detects the on a reboot booted kernel and patches that. If I patch a kernel the crossystem flags will not change immediately, because the patched kernel will only be active after a reboot. This could be tricky, I need to think about it. |
OK, it may be worth doing to avoid breaking an already-modified install, but it can't be used to save people about to nuke themselves. |
I would rather prefer something like @drinkcat proposed in #785 to be really sure all kernels are signed correctly before enabling the flag. We now also have the waiting time when the flag is disabled. On my HP chromebook 14 the test takes about 5-6 seconds. That said keeping |
OK, I had the opportunity to perform an update. I ran the update, then rebooted. Then, before I patched, I ran: echo echo After patching, I ran: echo echo After reboot, I ran: echo echo |
...oh well. |
Over 2 years later I just had this exact issue so it seems it is still not fixed |
crouton's current logic is to set 'dev_boot_signed_only=1' only when 'dev_boot_usb' and/or 'dev_boot_legacy' are not set / off. I have my options configured to boot both usb & legacy so crouton leaves the 'dev_boot_signed_only' bit unset, see below:
Hope this helps, |
@DennisLfromGA a few weeks ago, there was a known issue where certain older Toshiba Chromebook 2's lost their video on all channels except stable. So I reverted to stable to get video back, and re-installed crouton, but the command for targets did not work: sh ~/Downloads/crouton -t list I was fine installing cli-extra, but that should get fixed. |
It still works for me with or without
Hope this helps, P.S. But if it is broken it's probably in the 'pie' category so go for it! |
Thanks, it works, I forgot the "sh"! A suggestion would be to make crouton executable (600 or 666) and not need the sh ? |
The stateful_partition where your I re-mount it in my
Hope this helps, |
@DennisLfromGA Thanks! I'll test it when I'm home. I'm pretty sure I've chmodded scripts and they've executed in ~/Downloads, and I don't recall having changed the mount flags. These days, I'm hesitant to make many changes, since I seem to have to powerwash often. My first gen Toshiba Chromebook 2 is getting old, it still plays netflix fine, browses web, and runs crouton, but I don't think it's ever getting the ability to run android apps. |
I have been playing with Windows bash dev edition lately, and it needs crouton badly. Perhaps the contributors should sell it to MS for $100 Billion and evenly split the proceeds ?? ;) |
That work nicely |
I have an Acer C720 running ChromeOS 34.0.1847.118. I installed crouton on my Chromebook and it worked great. Because I wanted to use virtualization, I chose to add the disablevmx=off option to the kernel boot command line, which required that I set dev_boot_signed_only=0. All was well. Then I upgraded crouton, which set dev_boot_signed_only=1 again. When I rebooted, my Chromebook was unable to boot the active image, since it requires a kernel signed by Google.
I would like to suggest that when performing updates, crouton first check if the currently active image is signed by Google. If not, it should not set dev_boot_signed_only to 1. Otherwise people will run into the same problem as me. That should help for the future.
Does anyone know how I can either set dev_boot_signed_only=0 or boot using the previous boot image in /dev/sda2 or /dev/sda4? I believe the other boot partition has a boot image that IS signed by Google, so if I can get in, then I can run crossystem dev_boot_signed_only=0 and all should be well. It's just getting in that's the hard part. I'd prefer not to reformat the drive since I spent a couple of days getting the chroot Debian and the Windows VM under VirtualBox just so, and I'm not looking forward to repeating all that work.
The text was updated successfully, but these errors were encountered: