-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Talos II - 0.6.1 - Workstation - No USB keyboard support and no terminal redirection (expected withFBwhiptail) #193
Comments
@SergiiDmytruk : The following should fix the current issue linuxboot/heads@master...tlaurion:heads:talos2_cryptsetup2_usb_keyboard_for_workstation#diff-621bfa30b71e6d738181d2efc0cb94da0922a89dfc6831656518b44a17d62a1aR40-R42 Which enables usb as early as possible in init: https://github.com/osresearch/heads/blob/master/initrd/init#L82-L85 And where insmod wrapper measures and extends PCR prior of actually loading the module: https://github.com/osresearch/heads/blob/master/initrd/sbin/insmod I did not test USB keyboard on Talos II Workstation with those changes yet. Please reuse proposed changes on first link as needed. |
I thought there was some other option and only now noticed that it's the same |
No, as said over matrix off-channel discussion, even recompiling with usbhid in-kernel doesn't change the behavior for workstation. This is why it was said that behavior seems to be a kernel bad behavior. What is weird is that the server boad configuration receives input with usbhid compiled as module, as expected, where workstation doesn't. I thought workstation didn't load usbhid module and was wrong. It does. But still doesn't interact with USB keyboard on my side. It does on yours? So the insight is that its not the kernel module nor heads added logic, but something along framebuffer (the difference between server and workstation boards) where kernel options which made kgpe-d16 server and workstation different cannot be passed in coreboot config and defined in skiboot. Kgpe-d16 workstation board with USB keyboard works. Any update on testing corresponding options from skiboot? If that fix the behavior, then coreboot needs to be able to accept those so that a different coreboot config can be made for server and workstation, so that a dual console can be configured. Please take a look at kgpe-d16 coreboot configuration and passed kernel options. |
I also see:
And the behaviour seems similar to yours: in BMC KVM keyboard doesn't work until recovery is selected via serial console. However, there is no usbhid error for the hardware keyboard:
Any option specified there can be specified in
I did try making coreboot's |
With Linux v5.10.5 shows no difference compared to v5.5. Linux v6.0.1 behaves the same in my tests, but |
Last console specified has impact for input https://tldp.org/HOWTO/Remote-Serial-Console-HOWTO/configure-kernel.html Will re-read, had to deal with that with kgpe-d16/qemu boards) |
What might be happening here is that you try to set this manually while none of Linux payload nor LinuxBoot payload is selected first in the configuration. What we have is |
Are we sure tty0 is created on POWER systems? Kernel documentation says hvc0 is for POWER machines: |
dmesg console output from Heads
|
I've also added
to |
Resuing it will not work IIRC. |
Saw that it's already defined by two payloads and definitions are identical, copy&pasting that option made it work. I guess two lines above just don't constitute a proper definition. Image with |
@SergiiDmytruk Could you link here a branch/commit/CircleCI build so I know what i'm testing? Ideally to speed up/entice testing process, it would be nice if those were including a working branch link, so I can test a bunch of fixes all at once, lowering the time needed to test and also easying builds testing from the community we are trying to have test the builds and report maybe other issues i'm seldom reporting for the moment. As of now, I understand that
You want me to test those separately/conjointly (doesn't seem to make sense). |
Can be tested at the same time, one shouldn't affect the other. coreboot image is the latest release + patch to make
It's a build of linuxboot/heads#1222
The behaviour might be different in some way, I'm not sure. If keyboard still doesn't work, then |
I should add that |
Not sure I follow. So coreboot passes the kernel boot arguments in your coreboot image above (this ticket). This overrides the zImage.bundle arguments? so coreboot image should be tested to see if dmesg removes powersave (to validate). |
If
It seems to work ( src/mainboard/raptor-cs/talos-2/Kconfig | 6 ++++++
src/soc/ibm/power9/chip.c | 11 +++++++++++
2 files changed, 17 insertions(+)
diff --git a/src/mainboard/raptor-cs/talos-2/Kconfig b/src/mainboard/raptor-cs/talos-2/Kconfig
index ca662f2eab..d2d8273226 100644
--- a/src/mainboard/raptor-cs/talos-2/Kconfig
+++ b/src/mainboard/raptor-cs/talos-2/Kconfig
@@ -89,4 +89,10 @@ config DRAM_SIZE_MB
int
default 32768
+config LINUX_COMMAND_LINE
+ string "Linux command line"
+ default ""
+ help
+ A command line to add to the Linux kernel.
+
endif # BOARD_RAPTOR_CS_TALOS_2
diff --git a/src/soc/ibm/power9/chip.c b/src/soc/ibm/power9/chip.c
index 88c08af839..f4769b45fb 100644
--- a/src/soc/ibm/power9/chip.c
+++ b/src/soc/ibm/power9/chip.c
@@ -343,6 +343,16 @@ static void add_reserved_memory_node(struct device_tree *tree, uint64_t start, u
dt_add_reg_prop(node, &start, &size, 1, 2, 2);
}
+static void add_cmdline(struct device_tree *tree)
+{
+#ifdef CONFIG_LINUX_COMMAND_LINE
+ struct device_tree_node *node;
+
+ node = dt_find_node_by_path(tree, "/chosen", NULL, NULL, 1);
+ dt_add_string_prop(node, "bootargs", (const char *)CONFIG_LINUX_COMMAND_LINE);
+#endif
+}
+
static void add_memory_nodes(struct device_tree *tree)
{
struct mem_map map;
@@ -576,6 +586,7 @@ static int dt_platform_update(struct device_tree *tree, uint8_t chips)
validate_dt(tree, chips);
+ add_cmdline(tree);
add_memory_nodes(tree);
add_dimm_sensor_nodes(tree, chips);
add_cb_fdt_data(tree); Will need to merge it to have it in the next release. |
Ok, so I understand that zImage.bundled is the next step here on top of linuxboot/heads#1222 Doing:
Disconnects.
Falling back to server board artifacts of CircleCI to test linuxboot/heads#1222 |
Ok, as of current codebase: init calls cbfs-init, which for talos calls flashrom to get backup of the firmware (which currently reads it 3 times, so around 90 seconds, could be reduced to a minimum of 30) through https://github.com/osresearch/heads/pull/1222/files#diff-0102d43c07e95fc76c3c2fe1532c91a911cabcb96176d0895a937b13c8a7a5f4R10-R23 @SergiiDmytruk : since flashtools is already being modified, wouldn't it make more sense to modify cbfs to do the proper jumps if pnor and extract from ROM directly instead of relaying on flashrom to dump everything at each boot and rely on it (massive TOCTOU here on /tmp/? ) Otherwise, peek and poke from flashtools maybe? Adding 30 seconds at each boot (if flash.sh is modified to read only once, while write implies a verification) is kinda unacceptable here... |
Going back to the original USB keyboard issue. The behaviour on Hostboot is the same. So it is not a firmware issue. We have tried many different console combinations already. Initially the expectation was that we have a kernel module issue, while we might be hitting some limitation in the kernel console subsystem or need to know a specific workaround for making USB keyboards work when graphics is used. We are really out of ideas here and we do not see how we can support this from a firmware perspective. Maybe we should look for help in some POWER9 / Talos II communities, as this problem is not really Dasharo-specific. |
I just discovered that on the server board, Heads is answering commands through two different terminals, giving whiptail output on two different terminals individually for bmc/host, where the keyboard interacts with the host terminal and where the bmc keyboarfd interacts with bmc terminal only. Not sure why, but on kgpe-d16 board (older kernel as well) that was not possible through ast2050. The BMC and keyboard were interacting with the same terminal. This is problematic, since a local user could steal unsealed secrets under /tmp from a concurrent session. Heads expects to be single user on the machine... Any idea where that behavior is coming from? Intuition tells that if we had only one terminal where bmc/usb keyboard was directed to, the problem we have here (and the security issue) would vanish at the same time. Any insight welcome. Test by youself on server board: you can interact in parallel through bmc emulated keyboard-> bmc terminal/usb keyboard->VGA output. Not normal indeed. Issue discovered trying to tune terminal under linuxboot/heads#1237 |
This is how BMC terminal was made to work, by starting Heads for each console, same as Petitboot does it. I suspect that not passing two consoles to the kernel will just leave you without BMC or normal console and I'm not sure two consoles can be somehow synchronized to act as one (in a screen/tmux-like way). |
"When multiple consoles are listed output is sent to all consoles and input is taken from the last listed console. The last console is the one Linux uses as the /dev/console device." Right now default console is setuped to be "console=hvc0" being the last one targeted. Comparing kgpe-d16 config to talos II, the big difference we can see is that the first console is serial A dual console input/output is given with qemu board to the payload: Will try to build and test on kgpe-d16 and check with qemu dual input/output config to validate behavior there as well. |
I had to swap the order of consoles in |
No. Test the expected behavior with the following patch:
@SergiiDmytruk : As a result, you will get expected pause_recovery (etc/functions) output "Hit enter to proceed to recovery shell:" prompt on ttyS0 (stdio) while whiptail is shown only once on host (qemu/kvm). This is important in case of TPM, since entering recovery shell will invalidate measurements and prevent TPM disk encryption key to be unsealed. By removing the CONFIG_BOOT_RECOVERY_SERIAL board configuration export, then agreed, only dmesg is redirected to ttyS0, where again, last console passed per coreboot/skiboot/kernel configured boot option will be /dev/console, so of course switching them results into where the input output is being redirected/taken from. Same should apply for kgpe-d16 per board configs:
I am still unclear, since I was not able to build coreboot 4.11 for kgpe-d16 (race condition error with car.data at build time...) of what is the behavior there, where on kgpe-d16, ttyS1 is the output for console and where ttyS0 was the possible additional input, I do not recall exactly the details of how the BMC interaction was different and as I told to @miczyg1 in last community meeting, maybe my memory is failing me and the BMC was not able to control host's VGA fbwiptail as it is currently the case for Talos II. Not sure what is exactly the differences at play with hvc0 magic and cannot retest as of now behavior on kgpe-d16. Can you remind me what was wrong with stty not working for Talos and required agetty and spawning multiple whiptail on each available consoles? |
kgpe-d16 server:
Which tells us that with the help of CONFIG_BOOT_RECOVERY_SERIAL and dual output (tty0) and BMC(ttyS1) it is expected to have dual output and input: But the question of if remote bmc input was interacting with whiptail or not is not yet clear. |
I don't remember trying
|
@SergiiDmytruk things are unfolding themselves with tests of https://app.circleci.com/pipelines/github/tlaurion/heads?branch=talos_CONFIG_BOOT_RECOVERY_SERIAL builds. First things first, it is impossible to have multiple fbwhiptail output into anything else then the framebuffer itself. The problem we have with fbwhiptail as opposed to whiptail output is that even if CONFIG_BOOT_RECOVERY_SERIAL defines hvc0 to spawn a recovery console on hvc0 (bmc) anything that will be drawn on hvc0 outside of console output will attempt to output on fb, which hvc0 can't do. As in those test roms for workstation, hvc0 will present a console only recovery prompt (asking to type enter on hvc0 to spwan recovery shell and invalidate measurements), where tty0 shows fbwhiptail output. In TPM usage use case, anything going into the recovery shell will invalidate TPM measurements by extending https://osresearch.net/Keys/#tpm-pcrs PCR4. Same applies to server board in current testing ROMs from this PR, which shares exactly the same inversed config. The conclusion here are:
Unanswered questions:
My security related question before was related to a race condition and security hazard of having multiple consoles under Heads under TPM use case, unsealing secrets under /tmp and having those accessible after having been unsealed from TPM. The choice here is user experience based. Basically, my question for the next step of this would be to agree that either:
Thoughts? |
When using fbwhiptail, the content of bin/whiptail is replaced by a shell script that calls fbwhiptail instead, so the rest of the codebase simply calls whiptail, indifferenciating if using whiptail or fbwhiptail... So having both fbwhiptail and whiptail output would require more work. Basically, if we are to keep a workstation board that uses fbwhiptail today as currently delivered, it simply doesn't work for hvc0 for everything outputted to serial (ttyS0/hvc0). So the get home message here is that fbwhiptail enablement cannot be used in BMC use case as of now. We could implement a mitigation under bin/whiptail script to simply bail if $(tty) returns anything ttyS*/hv0 prior of attempting to call fbwhiptail as of now. But as current, fbwhiptail is damn slow (fb is not optimized somehow on used kernel) and screen is somehow corrupted (fuzzy). The good news is that USB keyboard support is now figured out on why it didn't work cannot spawn fbwhptail on hvc0, which is happening now and intercepts usb input somehow). |
Nice. Your explanation makes sense and explains why when 2 I guess I would pick two text consoles and one board. Doing changes for |
Exit criteria:
|
This is fixed under Heads since talos_2 unique board (whiptail) replacing fbwhiptail that was faulty and to be honest too slow to be useable. linuxboot/heads#1353 also fixed an accidental removal of agetty. I think this issue can be closed. |
Dasharo version
0.6
Dasharo variant
Workstation
Affected component(s) or functionality
Brief summary
USB keyboard doesn't work and there is no fbwhiptail redirection to bmc terminal (normal).
User has to use external laptop to control remotely over BMC yo control and see what is on connected over VGA port until final OS boots.
How reproducible
100%
How to reproduce
Boot workstation, type on usb keyboard to realise that nothing can be typed on keyboard to Heads.
Expected behavior
Have USB keyboard working to control workstation from boot.
Actual behavior
Input can only be done through bmc emulated usb to control workstation.
Screenshots
Additional context
USB_KEYBOARD (HID driver) should probably be defined under board config and compiled as module inside of linux configuration.
Solutions you've tried
Using server variant to test 0.6.1 as of now
The text was updated successfully, but these errors were encountered: