Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Talos II: flashrom, flashtools, PNOR handling #1222

Merged
merged 6 commits into from
Nov 11, 2022

Conversation

SergiiDmytruk
Copy link
Contributor

@SergiiDmytruk SergiiDmytruk commented Oct 10, 2022

Power is different from x86 in several ways:

  • coreboot is only a part of flash chip (part of PNOR)
  • coreboot is stored in PNOR with ECC
  • even after extracting it, last 4 bytes aren't pointer to CBFS master record
  • I think firmware image isn't accessible in memory space to the host OS

flashtools module here uses 3mdeb's fork, will send changes there after some review.

With these changes settings are loaded on startup, settings can be stored from the menu.

By the way, scripts read ROM 3 times and reading 64 MiB from flash 3 times takes noticeable amount of time. Edit: fixed under #1230

@krystian-hebel
Copy link
Contributor

Regarding access from host OS, I think that nothing blocks accesses to PNOR through LPC space (0x80060300F0000000, + 64MB), but software would have to explicitly use cache inhibiting instructions. You would still get the image with ECC and would have to parse partition headers, though.

That being said, I think "the proper way" would be to use OPAL calls.

@SergiiDmytruk
Copy link
Contributor Author

I think that nothing blocks accesses to PNOR through LPC space (0x80060300F0000000, + 64MB)

I thought maybe it's reserved, but I probably confused it for HOMER.

but software would have to explicitly use cache inhibiting instructions

A user-space software can use such instructions on a mmap of /dev/mem?

I think "the proper way" would be to use OPAL calls.

Yes, that's what mtd driver does on PPC.

@krystian-hebel
Copy link
Contributor

A user-space software can use such instructions on a mmap of /dev/mem?

Fair point, these are hypervisor instructions so in some cases even OS wouldn't be able to do it. I guess this should be doable through storage control bits (WIMG) in page tables, but I haven't looked at that - it wasn't necessary for coreboot stuff 🤷

@tlaurion
Copy link
Collaborator

user@host:~/Downloads/flashrom$ wget https://output.circle-artifacts.com/output/job/075c9abd-a139-4af5-aa2c-b4e31a236619/artifacts/0/build/ppc64/talos-2_workstation/heads-talos-2_workstation-v0.2.0-1267-g73bc621.bootblock https://output.circle-artifacts.com/output/job/075c9abd-a139-4af5-aa2c-b4e31a236619/artifacts/0/build/ppc64/talos-2_workstation/heads-talos-2_workstation-v0.2.0-1267-g73bc621.rom https://output.circle-artifacts.com/output/job/075c9abd-a139-4af5-aa2c-b4e31a236619/artifacts/0/build/ppc64/talos-2_workstation/zImage.bundled
user@host:~/Downloads/flashrom$ sha256sum *
70fdd76d2cb5568d1bff67d6e5dd3161cb0808b050f56007530014ab6691586d  heads-talos-2_workstation-v0.2.0-1267-g73bc621.bootblock
593f31eedb08eabe73a37333818bcc0a3b7d61b56f96366b6d3388db1bfc1d76  heads-talos-2_workstation-v0.2.0-1267-g73bc621.rom
ac5eb89932199f5bfa5340ee82e99323b44c9d55a97e6a8ae6ab2ce6a871ee6f  zImage.bundled
user@host:~/Downloads$ rsync -ravczz --inplace --rsh='ssh -p2300' flashrom [email protected]:/tmp/images/

BMC:

root@talos:/tmp/images/flashrom# pflash  -F ../../talos.pnor -f -P HBB -p *.bootblock && pflash  -F ../../talos.pnor -f -P HBI -p *.rom && pflash  -F ../../talos.pnor -f -P BOOTKERNEL -p zImage.bundled && mboxctl --backend file:/tmp/talos.pnor
About to program "heads-talos-2_workstation-v0.2.0-1267-g73bc621.bootblock" at 0x00205000..0x0020c002 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
About to program "heads-talos-2_workstation-v0.2.0-1267-g73bc621.rom" at 0x00425000..0x00666200 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
About to program "zImage.bundled" at 0x022a1000..0x02e61158 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
SetBackend: Success
root@talos:/tmp/images/flashrom# obmcutil poweroff && obmcutil poweron && obmc-console-client


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 bootblock starting (log level: 7)...
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
FMAP: Found "FLASH" version 1.1 at 0x20000.
FMAP: base = 0x0 size = 0x200000 #areas = 4
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
CBFS: mcache @0xf8231000 built for 10 files, used 0x1f0 of 0x2000 bytes
CBFS: Found 'fallback/romstage' @0x80 size 0x124a1 in mcache @0xf823102c
BS: bootblock times (exec / console): total (unknown) / 2 ms


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 romstage starting (log level: 7)...
IPMI: romstage PNP BT 0xe4
Get BMC self test result...Function Not Implemented
Initializing IPMI BMC watchdog timer
IPMI BMC watchdog initialized and started.
Initializing FSI...
Initialized FSI (chips mask: 0x01)
Building MVPDs...
starting istep 8.1
starting istep 8.2
starting istep 8.3
starting istep 8.4
starting istep 8.9
Base epsilon values read from table:
 R_T[0] = 22
 R_T[1] = 22
 R_T[2] = 103
 W_T[0] = 5
 W_T[1] = 33
Scaled epsilon values based on +20 percent guardband:
 R_T[0] = 27
 R_T[1] = 27
 R_T[2] = 124
 W_T[0] = 6
 W_T[1] = 40
starting istep 8.10
starting istep 8.11
starting istep 9.2
starting istep 9.4
starting istep 9.6
starting istep 9.7
starting istep 10.1
starting istep 10.6
starting istep 10.10
starting istep 10.12
starting istep 10.13
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
MEMD partition has ECC
MEMD is in 0x03cef200 through 0x03cfb917
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 51
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 52
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 53
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D4
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D5
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D6
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D7
SPD @ 0x50
SPD: module type is DDR4
SPD: module part number is M393A1K43BB0-CRC    
SPD: banks 16, ranks 1, rows 16, columns 10, density 8192 Mb
SPD: device width 8 bits, bus width 64 bits
SPD: module size is 8192 MB (per channel)
starting istep 13.1
starting istep 13.2
starting istep 13.3
starting istep 13.4
starting istep 13.5
starting istep 13.6
starting istep 13.7
starting istep 13.8
starting istep 13.9
starting istep 13.10
CCS took 2 us (3 us timeout), 1 instruction(s)
CCS took 2 us (2 us timeout), 14 instruction(s)
RCD dump for I2C address 0x58:
0xf820fa30: 80 b3 40 42 30 00 00 00 02 01 00 03 cb e3 c0 0d  ..@B0...........
0xf820fa40: 00 00 39 00 00 00 00 00 00 00 07 00 00 00 00 00  ..9.............
starting istep 13.11
CCS took 2 us (7 us timeout), 2 instruction(s)
Write Leveling starting
CCS took 12 us (92 us timeout), 5 instruction(s)
Write Leveling done
Initial Pattern Write starting
CCS took 5 us (38 us timeout), 5 instruction(s)
Initial Pattern Write done
DQS alignment starting
CCS took 10 us (44 us timeout), 1 instruction(s)
DQS alignment done
Read Clock Alignment starting
CCS took 8 us (82 us timeout), 1 instruction(s)
Read Clock Alignment done
Read Centering starting
CCS took 36 us (120 us timeout), 1 instruction(s)
Read Centering done
Write Centering starting
CCS took 8297 us (11314 us timeout), 7 instruction(s)
Write Centering done
Coarse write/read starting
CCS took 5 us (24 us timeout), 1 instruction(s)
Coarse write/read done
starting istep 13.12
starting istep 13.13
starting istep 14.1
starting istep 14.2
starting istep 14.3
Initializing PEC0...
Initializing PEC1...
Initializing PEC2...
Initializing PHB0...
Initializing PHB1...
Initializing PHB2...
Initializing PHB3...
Initializing PHB4...
Initializing PHB5...
starting istep 14.4
starting istep 14.5
0xF000F = 221d104900008040
CBMEM:
IMD: root @ 0xffeff000 254 entries.
IMD: root @ 0xffefec00 62 entries.
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
CBFS: Found 'fallback/ramstage' @0x12580 size 0xcca9 in mcache @0xf8231080
BS: romstage times (exec / console): total (unknown) / 17 ms


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 ramstage starting (log level: 7)...
Enumerating buses...
Root Device scanning...
DD21, boot core: 18
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HCODE partition has ECC
HCODE is in 0x01a82200 through 0x01b821ff
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
OCC partition has ECC
OCC is in 0x03822200 through 0x039221ff
Base epsilon values read from table:
 R_T[0] = 22
 R_T[1] = 22
 R_T[2] = 103
 W_T[0] = 5
 W_T[1] = 33
Scaled epsilon values based on +20 percent guardband:
 R_T[0] = 27
 R_T[1] = 27
 R_T[2] = 124
 W_T[0] = 6
 W_T[1] = 40
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
WOFDATA partition has ECC
WOFDATA is in 0x039ea200 through 0x03c94ca7
Matching WOF tables section not found, disabling WOF
Safe mode freq = 1833412 kHZ, voltage = 711 mv
starting istep 15.2
starting istep 15.3
starting istep 15.4
Starting PM complex...
Attempting PGPE activation...
PGPE was activated successfully
Done starting PM complex
Activating OCC...
Done activating OCC
starting istep 16.1
XIVE configured, entering dead man loop
ASSERTION ERROR: file '(filenames not available on timeless builds)', line 404
starting istep 18.11
starting istep 18.12
CPU_CLUSTER: 0 enabled
PNP: 00e4.0 enabled
scan_bus: bus Root Device finished in 3075 msecs
done
BS: BS_DEV_ENUMERATE run times (exec / console): 3069 / 7 ms
Allocating resources...
Reading resources...
CPU_CLUSTER: 0 missing read_resources
Done reading resources.
CPU_CLUSTER: 0 missing set_resources
PNP: 00e4.0 00 <- [0x00000000e4 - 0x00000000e6] size 0x00000003 gran 0x00 io
Done setting resources.
Done allocating resources.
BS: BS_DEV_RESOURCES run times (exec / console): 0 / 1 ms
Enabling resources...
done.
Initializing devices...
PNP: 00e4.0 init
IPMI: PNP BT 0xe4
Get BMC self test result...Function Not Implemented
IPMI: Found man_id 0x6cb0000, prod_id 0x1000000
IPMI: Version 2.0
PNP: 00e4.0 init finished in 16 msecs
Devices initialized
BS: BS_DEV_INIT run times (exec / console): 16 / 1 ms
Finalize devices...
Devices finalized
Writing coreboot table at 0xffed9000
 0. 0000000000000000-00000000f8207fff: RAM
 1. 00000000f8208000-00000000f820ffff: RAMSTAGE
 2. 00000000f8210000-00000000f8ffffff: RAM
 3. 00000000f9000000-00000000f929bfff: RAMSTAGE
 4. 00000000f929c000-00000000ffed8fff: RAM
 5. 00000000ffed9000-00000000ffefffff: CONFIGURATION TABLES
 6. 00000000fff00000-00000001ff3fffff: RAM
 7. 00000001ff400000-00000001ffffffff: RESERVED
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
Wrote coreboot table at: 0xffed9000, 0x214 bytes, checksum 3d45
coreboot table: 556 bytes.
IMD ROOT    0. 0xffeff000 0x00001000
IMD SMALL   1. 0xffefe000 0x00001000
CONSOLE     2. 0xffede000 0x00020000
MEM INFO    3. 0xffedb000 0x000022b0
COREBOOT    4. 0xffed9000 0x00002000
IMD small region:
  IMD ROOT    0. 0xffefec00 0x00000400
  RO MCACHE   1. 0xffefea00 0x000001f0
  FMAP        2. 0xffefe920 0x000000e0
BS: BS_WRITE_TABLES run times (exec / console): 1 / 5 ms
CBFS: Found 'fallback/payload' @0x257c0 size 0x632f9 in mcache @0xffefeb84
Checking segment from ROM address 0xf8380000
Checking segment from ROM address 0xf838001c
Checking segment from ROM address 0xf8380038
Loading segment from ROM address 0xf8380000
  code (compression=1)
  New segment dstaddr 0x00000000 memsize 0x173c00 srcaddr 0xf8380054 filesize 0x632a5
Loading Segment: addr: 0x00000000 memsz: 0x0000000000173c00 filesz: 0x00000000000632a5
using LZMA
Loading segment from ROM address 0xf838001c
  BSS 0x00300000 (113776 byte)
Loading Segment: addr: 0x00300000 memsz: 0x000000000001bc70 filesz: 0x0000000000000000
it's not compressed!
Clearing Segment: addr: 0x0000000000300000 memsz: 0x000000000001bc70
Loading segment from ROM address 0xf8380038
  Entry Point 0x00002570
BS: BS_PAYLOAD_LOAD run times (exec / console): 297 / 4 ms
Jumping to boot code at 0x00002570(0xffed9000)
CBFS: Found '1-cpu.dtb' @0x1f700 size 0x24e7 in mcache @0xffefeb3c
[   11.293827376,5] OPAL v6.3-rc2-32-gfa060c2c starting...
[   11.293830953,7] initial console log level: memory 7, driver 5
[   11.293832963,6] CPU: P9 generation processor (max 4 threads/core)
[   11.293834717,7] CPU: Boot CPU PIR is 0x0048 PVR is 0x004e1201
[   11.293837226,7] OPAL table: 0x30108230 .. 0x30108810, branch table: 0x30002000
[   11.293840335,7] Assigning physical memory map table for nimbus
[   11.293842693,7] FDT: Parsing fdt @0xf90a55f8
[   11.294544950,6] CHIP: Initialised chip 0 from xscom@603fc00000000
[   11.294568822,6] P9 DD2.10 detected
[   11.294570416,5] CHIP: Chip ID 0000 type: P9N DD2.10
[   11.294572479,7] XSCOM: Base address: 0x603fc00000000
[   11.294581144,7] XSTOP: XSCOM addr = 0x5012000, FIR bit = 31
[   11.294583281,6] MFSI 0:0: Initialized
[   11.294584699,6] MFSI 0:2: Initialized
[   11.294586081,6] MFSI 0:1: Initialized
[   11.294639411,6] LPC: LPC[000]: Initialized
[   11.294640842,7] LPC: access via MMIO @0x6030000000000
[   11.294653702,7] LPC: Default bus on chip 0x0
[   11.294660588,7] CPU: New max PIR set to 0x57
[   11.294691164,6] MEM: parsing reserved memory from node /reserved-memory
[   11.294695479,7] HOMER: Init chip 0
[   11.294697868,7]   PBA BAR0 : 0x00000001ff400000
[   11.294699587,7]   PBA MASK0: 0x0000000000300000
[   11.294701278,7]   HOMER Image at 0x1ff400000 size 4MB
[   11.294704138,7]   PBA BAR2 : 0x00000001ff800000
[   11.294705846,7]   PBA MASK2: 0x0000000000700000
[   11.294707487,7]   OCC Common Area at 0x1ff800000 size 8MB
[   11.294709792,7] CPU: decrementer bits 56
[   11.294714351,6] CPU: CPU from DT PIR=0x0048 Server#=0x48 State=3
[   11.294719683,6] CPU:  4 secondary threads
[   11.294722706,6] CPU: CPU from DT PIR=0x004c Server#=0x4c State=3
[   11.294728245,6] CPU:  4 secondary threads
[   11.294731250,6] CPU: CPU from DT PIR=0x0050 Server#=0x50 State=3
[   11.294736699,6] CPU:  4 secondary threads
[   11.294739665,6] CPU: CPU from DT PIR=0x0054 Server#=0x54 State=3
[   11.294745142,6] CPU:  4 secondary threads
[   11.294793040,6] PLAT: AST SIO unavailable!
[   11.294805270,7] UART: Using LPC IRQ 4
[   11.296828795,5] PLAT: Detected Talos platform
[   11.296870288,5] PLAT: Detected BMC platform ast2500:openbmc
[   11.311892293,5] CPU: All 16 processors called in...
[   11.404785604,7] LPC: Routing irq 10, policy: 0 (r=1)
[   11.404786613,7] LPC: SerIRQ 10 using route 0 targetted at OPAL
[   12.307680280,5] HIOMAP: Negotiated hiomap protocol v2
[   12.307735969,5] HIOMAP: Block size is 4KiB
[   12.307771303,5] HIOMAP: BMC suggested flash timeout of 8s
[   12.307806393,5] HIOMAP: Flash size is 64MiB
[   12.307843672,5] HIOMAP: Erase granule size is 4KiB
[   14.215216355,5] FLASH: Found system flash: (unnamed) id:0
[   15.005943865,5] STB: secure boot not supported
[   15.005998348,5] STB: trusted boot not supported
[   15.006049498,7] LPC: Routing irq 4, policy: 0 (r=1)
[   15.006050436,7] LPC: SerIRQ 4 using route 1 targetted at OPAL
[   15.006174244,5] OCC: All Chip Rdy after 0 ms
[   15.315411700,3] STB: container NOT VERIFIED, resource_id=4 secureboot not yet initialized
[   16.433184875,3] STB: container NOT VERIFIED, resource_id=3 secureboot not yet initialized
[   17.112673684,3] CAPP: Error loading ucode lid. index=201d1

Disconnection from BMC at this point.

Reconnection:

root@talos:~# obmc-console-client 
[  318.122975] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[  318.123070] mm/pgtable-generic.c:50: bad pmd c00600c2802001be.
[  318.123141] mm/pgtable-generic.c:50: bad pmd c00600c2804001be.
[  319.595189] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[  319.595263] mm/pgtable-generic.c:50: bad pmd c00600c2802001be.
[  319.595338] mm/pgtable-generic.c:50: bad pmd c00600c2804001be.
[  319.641238] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[  319.641319] mm/pgtable-generic.c:50: bad pmd c00600c2802001be.
[  319.641411] mm/pgtable-generic.c:50: bad pmd c00600c2804001be.
!!!!! User requested recovery shell
!!!!! Starting recovery shell

Since on workstation with Dasharo/dasharo-issues#193 unresolved here, there is output on Workstation's VGA, but no usb keyboard support. So each keypress here with commit 73bc621 generates a bad pmd error, until Exit to recovery shell is selected from BMC's opened obmc-client-console, until recovery shell as obtained. From there, we can interact with Heads from BMC.

Basic testing:

~ # time flash.sh -r /tmp/backup.rom
Board talos-2_workstation detected, continuing...
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
real	1m 24.66s
user	0m 5.87s
sys	1m 17.82s

Agreed that flashrom -v should be used instead of flashrom -r 3 times. There is not any misconnection I am aware of that should not detect a bad read from -v that would be detected having a direct, internal connection here, unless something was also writing at the same time. Normal usage of flashrom being from menus, we can imply here that Heads is doing the right thing and reduce that 1m24 reading and verifying to something smaller.

Testing:

~ # diff /bin/flash.sh.orig /bin/flash.sh
--- /bin/flash.sh.orig
+++ /bin/flash.sh
@@ -101,18 +101,10 @@
 flash_rom() {
   ROM=$1
   if [ "$READ" -eq 1 ]; then
-    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}.1" \
+    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}" \
     || die "$ROM: Read failed"
-    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}.2" \
-    || die "$ROM: Read failed"
-    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}.3" \
-    || die "$ROM: Read failed"
-    if [ `sha256sum ${ROM}.[123] | cut -f1 -d ' ' | uniq | wc -l` -eq 1 ]; then
-      mv ${ROM}.1 $ROM
-      rm ${ROM}.[23]
-    else
-      die "$ROM: Read inconsistent"
-    fi
+    flashrom $CONFIG_FLASHROM_OPTIONS -v "${ROM}" \
+    || die "$ROM: Verification failed"
   else
     cp "$ROM" /tmp/${CONFIG_BOARD}.rom
     sha256sum /tmp/${CONFIG_BOARD}.rom
~ # time flash.sh -r /tmp/testing.rom
Board talos-2_workstation detected, continuing...
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Verifying flash... VERIFIED.
real	0m 53.15s
user	0m 0.62s
sys	0m 51.83s

I guess better safe then sorry here, but really not sure of a case where verifying the rom internally is really useful....

Testing will be continued with server board, since testing interactions from menus is not really possible on workstation as of now.
Continuing with testing of Dasharo/dasharo-issues#193

@tlaurion
Copy link
Collaborator

Coming back from Dasharo/dasharo-issues#193 to continue tests of flashrom/pnor addition under flashtools.

From my understanding, coreboot calls zImage.bundle with hardcoded boot options from .config compiled in zImage.bundle.

So just pflashing the server board's zImage.bundle should do.

Let's see....
Going under https://app.circleci.com/pipelines/github/SergiiDmytruk/heads/88/workflows/6b2ef291-f306-4e3f-a6d7-29986a3c54e6/jobs/1639 and downloading only the server's board zImage.bundle

user@host:~/Downloads/flashrom$ rm zImage.bundled 
user@host:~/Downloads/flashrom$ wget https://output.circle-artifacts.com/output/job/dcc37748-25b7-47cf-bbab-09f0e7dc1a70/artifacts/0/build/ppc64/talos-2_server/zImage.bundled
user@host:~/Downloads/flashrom$ sha256sum *
70fdd76d2cb5568d1bff67d6e5dd3161cb0808b050f56007530014ab6691586d  heads-talos-2_workstation-v0.2.0-1267-g73bc621.bootblock
593f31eedb08eabe73a37333818bcc0a3b7d61b56f96366b6d3388db1bfc1d76  heads-talos-2_workstation-v0.2.0-1267-g73bc621.rom
c183ba610b76d44a520d769f9c251ba1b46f00d639844031b9d3a98d41959ca2  zImage.bundled

So here we have bootblock and coreboot artifacts from workstation, and zImage.bundled from server. Let's push only the required changes with rsync over BMC:

user@host:~/Downloads/flashrom$ cd ..
user@host:~/Downloads$ rsync -ravczz --inplace --rsh='ssh -p2300' flashrom [email protected]:/tmp/images/
sending incremental file list
flashrom/
flashrom/zImage.bundled
sent 11,941,121 bytes  received 21,141 bytes  52,123.15 bytes/sec
total size is 14,488,410  speedup is 1.21
root@talos:/tmp/images/flashrom# pflash  -F ../../talos.pnor -f -P BOOTKERNEL -p zImage.bundled && mboxctl --backend file:/tmp/talos.pnor
About to program "zImage.bundled" at 0x022a1000..0x02e2a158 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
SetBackend: Success
root@talos:/tmp/images/flashrom# obmcutil poweron && obmc-console-client
st typ

coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 bootblock starting (log level: 7)...
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
FMAP: Found "FLASH" version 1.1 at 0x20000.
FMAP: base = 0x0 size = 0x200000 #areas = 4
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
CBFS: mcache @0xf8231000 built for 10 files, used 0x1f0 of 0x2000 bytes
CBFS: Found 'fallback/romstage' @0x80 size 0x124a1 in mcache @0xf823102c
BS: bootblock times (exec / console): total (unknown) / 2 ms


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 romstage starting (log level: 7)...
IPMI: romstage PNP BT 0xe4
Get BMC self test result...Function Not Implemented
Initializing IPMI BMC watchdog timer
IPMI BMC watchdog initialized and started.
Initializing FSI...
Initialized FSI (chips mask: 0x01)
Building MVPDs...
starting istep 8.1
starting istep 8.2
starting istep 8.3
starting istep 8.4
starting istep 8.9
Base epsilon values read from table:
 R_T[0] = 22
 R_T[1] = 22
 R_T[2] = 103
 W_T[0] = 5
 W_T[1] = 33
Scaled epsilon values based on +20 percent guardband:
 R_T[0] = 27
 R_T[1] = 27
 R_T[2] = 124
 W_T[0] = 6
 W_T[1] = 40
starting istep 8.10
starting istep 8.11
starting istep 9.2
starting istep 9.4
starting istep 9.6
starting istep 9.7
starting istep 10.1
starting istep 10.6
starting istep 10.10
starting istep 10.12
starting istep 10.13
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
MEMD partition has ECC
MEMD is in 0x03cef200 through 0x03cfb917
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 51
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 52
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 53
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D4
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D5
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D6
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D7
SPD @ 0x50
SPD: module type is DDR4
SPD: module part number is M393A1K43BB0-CRC    
SPD: banks 16, ranks 1, rows 16, columns 10, density 8192 Mb
SPD: device width 8 bits, bus width 64 bits
SPD: module size is 8192 MB (per channel)
starting istep 13.1
starting istep 13.2
starting istep 13.3
starting istep 13.4
starting istep 13.5
starting istep 13.6
starting istep 13.7
starting istep 13.8
starting istep 13.9
starting istep 13.10
CCS took 2 us (3 us timeout), 1 instruction(s)
CCS took 2 us (2 us timeout), 14 instruction(s)
RCD dump for I2C address 0x58:
0xf820fa30: 80 b3 40 42 30 00 00 00 02 01 00 03 cb e3 c0 0d  ..@B0...........
0xf820fa40: 00 00 39 00 00 00 00 00 00 00 07 00 00 00 00 00  ..9.............
starting istep 13.11
CCS took 2 us (7 us timeout), 2 instruction(s)
Write Leveling starting
CCS took 12 us (92 us timeout), 5 instruction(s)
Write Leveling done
Initial Pattern Write starting
CCS took 5 us (38 us timeout), 5 instruction(s)
Initial Pattern Write done
DQS alignment starting
CCS took 10 us (44 us timeout), 1 instruction(s)
DQS alignment done
Read Clock Alignment starting
CCS took 8 us (82 us timeout), 1 instruction(s)
Read Clock Alignment done
Read Centering starting
CCS took 37 us (120 us timeout), 1 instruction(s)
Read Centering done
Write Centering starting
CCS took 8610 us (11314 us timeout), 7 instruction(s)
Write Centering done
Coarse write/read starting
CCS took 5 us (24 us timeout), 1 instruction(s)
Coarse write/read done
starting istep 13.12
starting istep 13.13
starting istep 14.1
starting istep 14.2
starting istep 14.3
Initializing PEC0...
Initializing PEC1...
Initializing PEC2...
Initializing PHB0...
Initializing PHB1...
Initializing PHB2...
Initializing PHB3...
Initializing PHB4...
Initializing PHB5...
starting istep 14.4
starting istep 14.5
0xF000F = 221d104900008040
CBMEM:
IMD: root @ 0xffeff000 254 entries.
IMD: root @ 0xffefec00 62 entries.
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
CBFS: Found 'fallback/ramstage' @0x12580 size 0xcca9 in mcache @0xf8231080
BS: romstage times (exec / console): total (unknown) / 17 ms


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 ramstage starting (log level: 7)...
Enumerating buses...
Root Device scanning...
DD21, boot core: 18
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HCODE partition has ECC
HCODE is in 0x01a82200 through 0x01b821ff
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
OCC partition has ECC
OCC is in 0x03822200 through 0x039221ff
Base epsilon values read from table:
 R_T[0] = 22
 R_T[1] = 22
 R_T[2] = 103
 W_T[0] = 5
 W_T[1] = 33
Scaled epsilon values based on +20 percent guardband:
 R_T[0] = 27
 R_T[1] = 27
 R_T[2] = 124
 W_T[0] = 6
 W_T[1] = 40
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
WOFDATA partition has ECC
WOFDATA is in 0x039ea200 through 0x03c94ca7
Matching WOF tables section not found, disabling WOF
Safe mode freq = 1833412 kHZ, voltage = 711 mv
starting istep 15.2
starting istep 15.3
starting istep 15.4
Starting PM complex...
Attempting PGPE activation...
PGPE was activated successfully
Done starting PM complex
Activating OCC...
Done activating OCC
starting istep 16.1
XIVE configured, entering dead man loop
ASSERTION ERROR: file '(filenames not available on timeless builds)', line 404
starting istep 18.11
starting istep 18.12
CPU_CLUSTER: 0 enabled
PNP: 00e4.0 enabled
scan_bus: bus Root Device finished in 3071 msecs
done
BS: BS_DEV_ENUMERATE run times (exec / console): 3066 / 6 ms
Allocating resources...
Reading resources...
CPU_CLUSTER: 0 missing read_resources
Done reading resources.
CPU_CLUSTER: 0 missing set_resources
PNP: 00e4.0 00 <- [0x00000000e4 - 0x00000000e6] size 0x00000003 gran 0x00 io
Done setting resources.
Done allocating resources.
BS: BS_DEV_RESOURCES run times (exec / console): 0 / 1 ms
Enabling resources...
done.
Initializing devices...
PNP: 00e4.0 init
IPMI: PNP BT 0xe4
Get BMC self test result...Function Not Implemented
IPMI: Found man_id 0x6cb0000, prod_id 0x1000000
IPMI: Version 2.0
PNP: 00e4.0 init finished in 27 msecs
Devices initialized
BS: BS_DEV_INIT run times (exec / console): 27 / 1 ms
Finalize devices...
Devices finalized
Writing coreboot table at 0xffed9000
 0. 0000000000000000-00000000f8207fff: RAM
 1. 00000000f8208000-00000000f820ffff: RAMSTAGE
 2. 00000000f8210000-00000000f8ffffff: RAM
 3. 00000000f9000000-00000000f929bfff: RAMSTAGE
 4. 00000000f929c000-00000000ffed8fff: RAM
 5. 00000000ffed9000-00000000ffefffff: CONFIGURATION TABLES
 6. 00000000fff00000-00000001ff3fffff: RAM
 7. 00000001ff400000-00000001ffffffff: RESERVED
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
Wrote coreboot table at: 0xffed9000, 0x214 bytes, checksum 3d45
coreboot table: 556 bytes.
IMD ROOT    0. 0xffeff000 0x00001000
IMD SMALL   1. 0xffefe000 0x00001000
CONSOLE     2. 0xffede000 0x00020000
MEM INFO    3. 0xffedb000 0x000022b0
COREBOOT    4. 0xffed9000 0x00002000
IMD small region:
  IMD ROOT    0. 0xffefec00 0x00000400
  RO MCACHE   1. 0xffefea00 0x000001f0
  FMAP        2. 0xffefe920 0x000000e0
BS: BS_WRITE_TABLES run times (exec / console): 1 / 5 ms
CBFS: Found 'fallback/payload' @0x257c0 size 0x632f9 in mcache @0xffefeb84
Checking segment from ROM address 0xf8380000
Checking segment from ROM address 0xf838001c
Checking segment from ROM address 0xf8380038
Loading segment from ROM address 0xf8380000
  code (compression=1)
  New segment dstaddr 0x00000000 memsize 0x173c00 srcaddr 0xf8380054 filesize 0x632a5
Loading Segment: addr: 0x00000000 memsz: 0x0000000000173c00 filesz: 0x00000000000632a5
using LZMA
Loading segment from ROM address 0xf838001c
  BSS 0x00300000 (113776 byte)
Loading Segment: addr: 0x00300000 memsz: 0x000000000001bc70 filesz: 0x0000000000000000
it's not compressed!
Clearing Segment: addr: 0x0000000000300000 memsz: 0x000000000001bc70
Loading segment from ROM address 0xf8380038
  Entry Point 0x00002570
BS: BS_PAYLOAD_LOAD run times (exec / console): 302 / 4 ms
Jumping to boot code at 0x00002570(0xffed9000)
CBFS: Found '1-cpu.dtb' @0x1f700 size 0x24e7 in mcache @0xffef[   11.308576782,5] OPAL v6.3-rc2-32-gfa060c2c starting...
[   11.308580256,7] initial console log level: memory 7, driver 5
[   11.308582301,6] CPU: P9 generation processor (max 4 threads/core)
[   11.308584095,7] CPU: Boot CPU PIR is 0x0048 PVR is 0x004e1201
[   11.308586593,7] OPAL table: 0x30108230 .. 0x30108810, branch table: 0x30002000
[   11.308589660,7] Assigning physical memory map table for nimbus
[   11.308592057,7] FDT: Parsing fdt @0xf90a55f8
[   11.309293658,6] CHIP: Initialised chip 0 from xscom@603fc00000000
[   11.309317338,6] P9 DD2.10 detected
[   11.309318849,5] CHIP: Chip ID 0000 type: P9N DD2.10
[   11.309320853,7] XSCOM: Base address: 0x603fc00000000
[   11.309329497,7] XSTOP: XSCOM addr = 0x5012000, FIR bit = 31
[   11.309331591,6] MFSI 0:0: Initialized
[   11.309333002,6] MFSI 0:2: Initialized
[   11.309334387,6] MFSI 0:1: Initialized
[   11.309387639,6] LPC: LPC[000]: Initialized
[   11.309389037,7] LPC: access via MMIO @0x6030000000000
[   11.309402036,7] LPC: Default bus on chip 0x0
[   11.309408863,7] CPU: New max PIR set to 0x57
[   11.309438960,6] MEM: parsing reserved memory from node /reserved-memory
[   11.309443213,7] HOMER: Init chip 0
[   11.309445485,7]   PBA BAR0 : 0x00000001ff400000
[   11.309447180,7]   PBA MASK0: 0x0000000000300000
[   11.309448863,7]   HOMER Image at 0x1ff400000 size 4MB
[   11.309451689,7]   PBA BAR2 : 0x00000001ff800000
[   11.309453340,7]   PBA MASK2: 0x0000000000700000
[   11.309454964,7]   OCC Common Area at 0x1ff800000 size 8MB
[   11.309457280,7] CPU: decrementer bits 56
[   11.309461890,6] CPU: CPU from DT PIR=0x0048 Server#=0x48 State=3
[   11.309467223,6] CPU:  4 secondary threads
[   11.309470265,6] CPU: CPU from DT PIR=0x004c Server#=0x4c State=3
[   11.309475818,6] CPU:  4 secondary threads
[   11.309478824,6] CPU: CPU from DT PIR=0x0050 Server#=0x50 State=3
[   11.309484318,6] CPU:  4 secondary threads
[   11.309487341,6] CPU: CPU from DT PIR=0x0054 Server#=0x54 State=3
[   11.309492788,6] CPU:  4 secondary threads
[   11.309540048,6] PLAT: AST SIO unavailable!
[   11.309552342,7] UART: Using LPC IRQ 4
[   11.311581724,5] PLAT: Detected Talos platform
[   11.311623091,5] PLAT: Detected BMC platform ast2500:openbmc
[   11.326671624,5] CPU: All 16 processors called in...
[   11.419572236,7] LPC: Routing irq 10, policy: 0 (r=1)
[   11.419573300,7] LPC: SerIRQ 10 using route 0 targetted at OPAL
[   12.322469304,5] HIOMAP: Negotiated hiomap protocol v2
[   12.322524819,5] HIOMAP: Block size is 4KiB
[   12.322552428,5] HIOMAP: BMC suggested flash timeout of 8s
[   12.322600892,5] HIOMAP: Flash size is 64MiB
[   12.322640117,5] HIOMAP: Erase granule size is 4KiB
[   14.230017996,5] FLASH: Found system flash: (unnamed) id:0
[   15.020758254,5] STB: secure boot not supported
[   15.020810726,5] STB: trusted boot not supported
[   15.020850509,7] LPC: Routing irq 4, policy: 0 (r=1)
[   15.020851463,7] LPC: SerIRQ 4 using route 1 targetted at OPAL
[   15.020974276,5] OCC: All Chip Rdy after 0 ms
[   15.330215008,3] STB: container NOT VERIFIED, resource_id=4 secureboot not yet initialized
[   16.447987149,3] STB: container NOT VERIFIED, resource_id=3 secureboot not yet initialized
[   17.127499005,3] CAPP: Error loading ucode lid. index=201d1

Disconnects here.
Upon reconnection, bmc gets to console and moving arrows give the wiptail output on BMC console. Good.

Applying precedent patch

~ # patch -p1 < /tmp/patch 
patching file bin/flash.sh

Attempting injection of my public key

~ # gpg-gui.sh # Selecting "Add GPG key to running BIOS and reflash" with USB Thumb drive connected with public key on it, selecting /dev/sda1, then public key name from mounted /media drive, then accepting to reflash with updated version at prompt
gpg: //.gnupg/trustdb.gpg: trustdb created
gpg: key E7B4A71658E36A93: public key "Insurgo Technologies Libres / Open Technologies <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2023-04-20
Board talos-2_server detected, continuing...
9d1e2c87a576527423e78d4ec83f16240b09ee2f332ddd1c5940a8b2744d17a6  /tmp/talos-2_server.rom

Initializing internal Flash Programmer
Reading old flash contents. Please wait...

The flashing prompt on screen stays at 0%, leading to think that what is being written to flash is way more then just the added keyring and trustdb...
Then:

Verifying flash contents. Please wait...

Error flashing coreboot -- see timestampped flashrom log in /tmp for more info

/tmp/gpg-gui.rom: Flash failed

The log under /tmp/flashrom-xxxxx.log contains a log of strings, linked to dynamic output produced with flash.sh dynamic progress output. It ends with

Erase/write done.
Verifying flash... VERIFIED.

So let's see, I guess....

Issuing reboot through Heads with reboot

[ 1136.019573] sysrq: Emergency Sync
[ 1136.019876] sysrq: Emergency Remount R/O
[ 1136.020195] sysrq: Resetting
[ 1159.334224491,5] OPAL: Reboot request...
[ 1159.334292792,5] RESET: Initiating fast reboot 1...
[ 1159.397542436,5] Clearing unused memory:
[ 1159.397636685,5] PCI: Clearing all devices...
[ 1159.397636827,7] Clearing region 31ec0000-ffe00000
[ 1159.408790163,5] PCI: Resetting PHBs and training links...
[ 1161.199397603,5] PCI: Probing slots...
[ 1161.255864606,3] PCI: PHB  (80000) not found
[ 1161.256102294,3] PCI: PHB  (80001) not found
[ 1161.256283603,3] PCI: PHB  (80003) not found
[ 1161.256493307,5] PCI Summary:
[ 1161.256630081,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=CPU1 Slot2 (16x) 
[ 1161.256997891,5] PHB#0001:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=CPU1 Slot1 (8x) 
[ 1161.257372773,5] PHB#0002:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=Builtin SAS 
[ 1161.257784974,5] PHB#0003:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=Builtin USB 
[ 1161.258152795,5] PHB#0003:01:00.0 [EP  ] 104c 8241 R:02 C:0c0330 (      usb-xhci) LOC_CODE=Builtin USB
[ 1161.258498995,5] PHB#0004:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=Builtin Ethernet 
[ 1161.258959802,5] PHB#0004:01:00.0 [EP  ] 14e4 1657 R:00 C:020000 (      ethernet) LOC_CODE=Builtin Ethernet
[ 1161.259399201,5] PHB#0004:01:00.1 [EP  ] 14e4 1657 R:00 C:020000 (      ethernet) LOC_CODE=Builtin Ethernet
[ 1161.259781705,5] PHB#0005:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..02 SLOT=BMC 
[ 1161.260161561,5] PHB#0005:01:00.0 [ETOX] 1a03 1150 R:04 C:060400 B:02..02 LOC_CODE=BMC
[ 1161.260746630,5] PHB#0005:02:00.0 [PCID] 1a03 2000 R:41 C:030000 (           vga) LOC_CODE=BMC
[ 1161.261208066,5] Clearing memory... 3/7GB done
[ 1161.261428127,5] Clearing memory... 4/7GB done
[ 1161.261589459,5] Clearing memory... 7/7GB done
[ 1161.261786760,5] IPMI: Resetting boot count on successful boot
[ 1161.261982873,5] INIT: Waiting for kernel...
[ 1164.255014557,3] STB: container NOT VERIFIED, resource_id=0 secureboot not yet initialized
[ 1164.256682626,5] INIT: 64-bit LE kernel discovered
[ 1164.272129446,5] OCC: All Chip Rdy after 0 ms
[ 1164.288207064,5] INIT: Starting kernel at 0x20010000, fdt at 0x30762b40 128483 bytes

zImage starting: loaded at 0x0000000020010000 (sp: 0x0000000020b87ee0)
Allocating 0x27a3e80 bytes for kernel...
Decompressing (0x0000000000000000 <- 0x0000000020024000:0x0000000020b853b5)...
Done! Decompressed 0x25a98e0 bytes

Linux/PowerPC load: 
Finalizing device tree... flat tree at 0x20b88ce0
[ 1165.227247415,3] LPC[000]: Got SYNC no-response error. Error address reg: 0xd0010080
[ 1165.227260011,6] IPMI: dropping non severe PEL event
[    4.500810] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[    4.500813] mm/pgtable-generic.c:50: bad pmd c00600c2802001be.
[    4.500814] mm/pgtable-generic.c:50: bad pmd c00600c2804001be.
[    4.811750] kAFS: failed to register: -97
Reading ROM...
Board talos-2_server detected, continuing...
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... 
[.....]

Leads back to Heads menu, which gives different output. Seems like there is a public key in keyring.
Options->GPG Options->List GPG keys in keyring
Shows actual GPG keyring. So the above was console corruption.... Scary.

Let's redo without patch, again from recovery shell so we can have error output to console again, and without my patch.

gpg-gui.sh #-> Replace GPG keys in current ROM and reflash, proceed to prompt, fails again
Flashing: [\                                                ] (0%)
Verifying flash contents. Please wait...

Error flashing coreboot -- see timestampped flashrom log in /tmp for more info

/tmp/gpg-gui.rom: Flash failed
~ # echo $?
1
wc -l /tmp/flashrom-20221014-184646.log
14 /tmp/flashrom-20221014-184646.log
head -n 11 /tmp/flashrom-20221014-184646.log
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom was built with libpci 3.5.4, GCC 8.3.0, little endian
Command line (8 args): flashrom --noverify-all -p linux_mtd -w /tmp/talos-2_server.rom -V -o /tmp/flashrom-20221014-184646.log
Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Initializing linux_mtd programmer
get_mtd_info: device_name: "PNOR", is_writeable: 1, numeraseregions: 0, total_size: 67108864, erasesize: 4096
Opened /dev/mtd0 successfully
The following protocols are supported: Programmer-specific.
Probing for Programmer Opaque flash chip, 0 kB: Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linu
x_mtd.
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific).
Reading old flash chip contents... done.
~ # cat /etc/config
export CONFIG_BOARD_NAME="Talos 2 Server"
export CONFIG_BOOTSCRIPT="/bin/gui-init"
export CONFIG_BOOT_DEV="/dev/sda1"
export CONFIG_BOOT_EXTRA_TTYS="tty0"
export CONFIG_BOOT_KERNEL_ADD="console=tty0 console=hvc0 rootdelay=3 rootwait panic=10"
export CONFIG_BOOT_KERNEL_REMOVE="quiet"
export CONFIG_BOOT_REQ_HASH="n"
export CONFIG_BOOT_REQ_ROLLBACK="n"
export CONFIG_COREBOOT="y"
export CONFIG_COREBOOT_VERSION="talos_2"
export CONFIG_FLASHROM_OPTIONS="--noverify-all -p linux_mtd"
export CONFIG_LINUX_VERSION="5.5-openpower"
export CONFIG_TPM="y"
export CONFIG_USB_KEYBOARD="y"
export CONFIG_USE_AGETTY="y"
export GIT_HASH='73bc621133dd767dbfa3bfe81173401db4ca0d76'
export GIT_STATUS=clean
export CONFIG_BOARD=talos-2_server

Manually testing flashrom

~ # flashrom --noverify-all -p linux_mtd -w /tmp/talos-2_server.rom 
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading old flash chip contents... done.
Erasing and writing flash chip... 
Warning: Chip content is identical to the requested image.
Erase/write done.
~ # echo $?
0

Hmmm....

flashrom --noverify-all -p linux_mtd -w /tmp/gpg-gui.rom
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading old flash chip contents... done.
Erasing and writing flash chip... 
Warning: Chip content is identical to the requested image.
Erase/write done.
~ # echo $?
0

Ok... Will redo and review the scripts, not sure where flashrom exits 1

@tlaurion
Copy link
Collaborator

For good measure, since I think about it, output of cbeme and dmesg

~ # cbmem -1

coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 bootblock starting (log level: 7)...
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
FMAP: Found "FLASH" version 1.1 at 0x20000.
FMAP: base = 0x0 size = 0x200000 #areas = 4
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
CBFS: mcache @0xf8231000 built for 10 files, used 0x1f0 of 0x2000 bytes
CBFS: Found 'fallback/romstage' @0x80 size 0x124a1 in mcache @0xf823102c
BS: bootblock times (exec / console): total (unknown) / 2 ms


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 romstage starting (log level: 7)...
IPMI: romstage PNP BT 0xe4
Get BMC self test result...Function Not Implemented
Initializing IPMI BMC watchdog timer
IPMI BMC watchdog initialized and started.
Initializing FSI...
Initialized FSI (chips mask: 0x01)
Building MVPDs...
starting istep 8.1
starting istep 8.2
starting istep 8.3
starting istep 8.4
starting istep 8.9
Base epsilon values read from table:
 R_T[0] = 22
 R_T[1] = 22
 R_T[2] = 103
 W_T[0] = 5
 W_T[1] = 33
Scaled epsilon values based on +20 percent guardband:
 R_T[0] = 27
 R_T[1] = 27
 R_T[2] = 124
 W_T[0] = 6
 W_T[1] = 40
starting istep 8.10
starting istep 8.11
starting istep 9.2
starting istep 9.4
starting istep 9.6
starting istep 9.7
starting istep 10.1
starting istep 10.6
starting istep 10.10
starting istep 10.12
starting istep 10.13
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
MEMD partition has ECC
MEMD is in 0x03cef200 through 0x03cfb917
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 51
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 52
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address 53
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D4
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D5
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D6
I2C transfer failed to complete (0x04011f0104000000)
No memory DIMM at address D7
SPD @ 0x50
SPD: module type is DDR4
SPD: module part number is M393A1K43BB0-CRC    
SPD: banks 16, ranks 1, rows 16, columns 10, density 8192 Mb
SPD: device width 8 bits, bus width 64 bits
SPD: module size is 8192 MB (per channel)
starting istep 13.1
starting istep 13.2
starting istep 13.3
starting istep 13.4
starting istep 13.5
starting istep 13.6
starting istep 13.7
starting istep 13.8
starting istep 13.9
starting istep 13.10
CCS took 2 us (3 us timeout), 1 instruction(s)
CCS took 2 us (2 us timeout), 14 instruction(s)
RCD dump for I2C address 0x58:
0xf820fa30: 80 b3 40 42 30 00 00 00 02 01 00 03 cb e3 c0 0d  ..@B0...........
0xf820fa40: 00 00 39 00 00 00 00 00 00 00 07 00 00 00 00 00  ..9.............
starting istep 13.11
CCS took 2 us (7 us timeout), 2 instruction(s)
Write Leveling starting
CCS took 12 us (92 us timeout), 5 instruction(s)
Write Leveling done
Initial Pattern Write starting
CCS took 5 us (38 us timeout), 5 instruction(s)
Initial Pattern Write done
DQS alignment starting
CCS took 10 us (44 us timeout), 1 instruction(s)
DQS alignment done
Read Clock Alignment starting
CCS took 8 us (82 us timeout), 1 instruction(s)
Read Clock Alignment done
Read Centering starting
CCS took 37 us (120 us timeout), 1 instruction(s)
Read Centering done
Write Centering starting
CCS took 8610 us (11314 us timeout), 7 instruction(s)
Write Centering done
Coarse write/read starting
CCS took 5 us (24 us timeout), 1 instruction(s)
Coarse write/read done
starting istep 13.12
starting istep 13.13
starting istep 14.1
starting istep 14.2
starting istep 14.3
Initializing PEC0...
Initializing PEC1...
Initializing PEC2...
Initializing PHB0...
Initializing PHB1...
Initializing PHB2...
Initializing PHB3...
Initializing PHB4...
Initializing PHB5...
starting istep 14.4
starting istep 14.5
0xF000F = 221d104900008040
CBMEM:
IMD: root @ 0xffeff000 254 entries.
IMD: root @ 0xffefec00 62 entries.
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
CBFS: Found 'fallback/ramstage' @0x12580 size 0xcca9 in mcache @0xf8231080
BS: romstage times (exec / console): total (unknown) / 17 ms


coreboot--TIMELESS--LESSTIME--Heads-v0.2.0-1267-g73bc621 Thu Jan  1 00:00:00 UTC 1970 ramstage starting (log level: 7)...
Enumerating buses...
Root Device scanning...
DD21, boot core: 18
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HCODE partition has ECC
HCODE is in 0x01a82200 through 0x01b821ff
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
OCC partition has ECC
OCC is in 0x03822200 through 0x039221ff
Base epsilon values read from table:
 R_T[0] = 22
 R_T[1] = 22
 R_T[2] = 103
 W_T[0] = 5
 W_T[1] = 33
Scaled epsilon values based on +20 percent guardband:
 R_T[0] = 27
 R_T[1] = 27
 R_T[2] = 124
 W_T[0] = 6
 W_T[1] = 40
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
WOFDATA partition has ECC
WOFDATA is in 0x039ea200 through 0x03c94ca7
Matching WOF tables section not found, disabling WOF
Safe mode freq = 1833412 kHZ, voltage = 711 mv
starting istep 15.2
starting istep 15.3
starting istep 15.4
Starting PM complex...
Attempting PGPE activation...
PGPE was activated successfully
Done starting PM complex
Activating OCC...
Done activating OCC
starting istep 16.1
XIVE configured, entering dead man loop
ASSERTION ERROR: file '(filenames not available on timeless builds)', line 404
starting istep 18.11
starting istep 18.12
CPU_CLUSTER: 0 enabled
PNP: 00e4.0 enabled
scan_bus: bus Root Device finished in 3071 msecs
done
BS: BS_DEV_ENUMERATE run times (exec / console): 3066 / 6 ms
Allocating resources...
Reading resources...
CPU_CLUSTER: 0 missing read_resources
Done reading resources.
CPU_CLUSTER: 0 missing set_resources
PNP: 00e4.0 00 <- [0x00000000e4 - 0x00000000e6] size 0x00000003 gran 0x00 io
Done setting resources.
Done allocating resources.
BS: BS_DEV_RESOURCES run times (exec / console): 0 / 1 ms
Enabling resources...
done.
Initializing devices...
PNP: 00e4.0 init
IPMI: PNP BT 0xe4
Get BMC self test result...Function Not Implemented
IPMI: Found man_id 0x6cb0000, prod_id 0x1000000
IPMI: Version 2.0
PNP: 00e4.0 init finished in 27 msecs
Devices initialized
BS: BS_DEV_INIT run times (exec / console): 27 / 1 ms
Finalize devices...
Devices finalized
Writing coreboot table at 0xffed9000
 0. 0000000000000000-00000000f8207fff: RAM
 1. 00000000f8208000-00000000f820ffff: RAMSTAGE
 2. 00000000f8210000-00000000f8ffffff: RAM
 3. 00000000f9000000-00000000f929bfff: RAMSTAGE
 4. 00000000f929c000-00000000ffed8fff: RAM
 5. 00000000ffed9000-00000000ffefffff: CONFIGURATION TABLES
 6. 00000000fff00000-00000001ff3fffff: RAM
 7. 00000001ff400000-00000001ffffffff: RESERVED
FMAP: area COREBOOT found @ 20200 (1965568 bytes)
FFS header at 0x80060300ffff7000
PNOR base at 0x80060300fc000000
HBI partition has ECC
HBI is in 0x00426200 through 0x0175f037
Wrote coreboot table at: 0xffed9000, 0x214 bytes, checksum 3d45
coreboot table: 556 bytes.
IMD ROOT    0. 0xffeff000 0x00001000
IMD SMALL   1. 0xffefe000 0x00001000
CONSOLE     2. 0xffede000 0x00020000
MEM INFO    3. 0xffedb000 0x000022b0
COREBOOT    4. 0xffed9000 0x00002000
IMD small region:
  IMD ROOT    0. 0xffefec00 0x00000400
  RO MCACHE   1. 0xffefea00 0x000001f0
  FMAP        2. 0xffefe920 0x000000e0
BS: BS_WRITE_TABLES run times (exec / console): 1 / 5 ms
CBFS: Found 'fallback/payload' @0x257c0 size 0x632f9 in mcache @0xffefeb84
Checking segment from ROM address 0xf8380000
Checking segment from ROM address 0xf838001c
Checking segment from ROM address 0xf8380038
Loading segment from ROM address 0xf8380000
  code (compression=1)
  New segment dstaddr 0x00000000 memsize 0x173c00 srcaddr 0xf8380054 filesize 0x632a5
Loading Segment: addr: 0x00000000 memsz: 0x0000000000173c00 filesz: 0x00000000000632a5
using LZMA
Loading segment from ROM address 0xf838001c
  BSS 0x00300000 (113776 byte)
Loading Segment: addr: 0x00300000 memsz: 0x000000000001bc70 filesz: 0x0000000000000000
it's not compressed!
Clearing Segment: addr: 0x0000000000300000 memsz: 0x000000000001bc70
Loading segment from ROM address 0xf8380038
  Entry Point 0x00002570
BS: BS_PAYLOAD_LOAD run times (exec / console): 302 / 4 ms
Jumping to boot code at 0x00002570(0xffed9000)
CBFS: Found '1-cpu.dtb' @0x1f700 size 0x24e7 in mcache @0xffefeb3c

~ # dmesg
[    0.000000] dt-cpu-ftrs: setup for ISA 3000
[    0.000000] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
[    0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x0001c86f8f5fb1a7 0x3c006041
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] radix-mmu: Activating Kernel Userspace Execution Prevention
[    0.000000] radix-mmu: Activating Kernel Userspace Access Prevention
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000002600000 with 2.00 MiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000002600000-0x0000000040000000 with 2.00 MiB pages
[    0.000000] radix-mmu: Mapped 0x0000000040000000-0x00000000c0000000 with 1.00 GiB pages
[    0.000000] radix-mmu: Mapped 0x00000000c0000000-0x00000000ffe00000 with 2.00 MiB pages
[    0.000000] radix-mmu: Mapped 0x00000000fff00000-0x0000000100000000 with 64.0 KiB pages
[    0.000000] radix-mmu: Mapped 0x0000000100000000-0x00000001c0000000 with 1.00 GiB pages
[    0.000000] radix-mmu: Mapped 0x00000001c0000000-0x00000001ff400000 with 2.00 MiB pages
[    0.000000] radix-mmu: Initializing Radix MMU
[    0.000000] Linux version 5.5.0-openpower1 (linux-talos-2_server.config@linuxboot) (gcc version 8.3.0 (GCC)) #0 SMP 1970-00-00
[    0.000000] OPAL: Found memory mapped LPC bus on chip 0
[    0.000000] ISA: Non-PCI bridge is /lpcm-opb@6030000000000/lpc@0
[    0.000000] Using PowerNV machine description
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] CPU maps initialized for 4 threads per core
[    0.000000]  (thread shift is 2)
[    0.000000] Allocated 3072 bytes for 16 pacas
[    0.000000] -----------------------------------------------------
[    0.000000] phys_mem_size     = 0x1ff300000
[    0.000000] dcache_bsize      = 0x80
[    0.000000] icache_bsize      = 0x80
[    0.000000] cpu_features      = 0x0001c86f8f5fb1a7
[    0.000000]   possible        = 0x0001fbefcf5fb1a7
[    0.000000]   always          = 0x00000003800081a1
[    0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[    0.000000] mmu_features      = 0xbc006041
[    0.000000] firmware_features = 0x0000000010000000
[    0.000000] vmalloc start     = 0xc008000000000000
[    0.000000] IO start          = 0xc00a000000000000
[    0.000000] vmemmap start     = 0xc00c000000000000
[    0.000000] -----------------------------------------------------
[    0.000000] numa:   NODE_DATA [mem 0x1ff1ef280-0x1ff1f3fff]
[    0.000000] rfi-flush: fallback displacement flush available
[    0.000000] rfi-flush: patched 5 locations (fallback displacement flush)
[    0.000000] link-stack-flush: software flush disabled.
[    0.000000] count-cache-flush: software flush disabled.
[    0.000000] stf-barrier: eieio barrier available
[    0.000000] stf-barrier: patched 57 entry locations (eieio barrier)
[    0.000000] stf-barrier: patched 5 exit locations (eieio barrier)
[    0.000000] Initializing IODA2 PHB (/pciex@600c3c0000000)
[    0.000000] PCI host bridge /pciex@600c3c0000000 (primary) ranges:
[    0.000000]  MEM 0x000600c000000000..0x000600c07ffeffff -> 0x0000000080000000 
[    0.000000] ioremap() called early from pnv_pci_init_ioda_phb+0x388/0xc90. Use early_ioremap() instead
[    0.000000]  MEM 0x0006000000000000..0x0006003fffffffff -> 0x0006000000000000 (M64 #1..31)
[    0.000000]  Using M64 #31 as default window
[    0.000000]   512 (511) PE's M32: 0x80000000 [segment=0x400000]
[    0.000000]                  M64: 0x4000000000 [segment=0x20000000]
[    0.000000]   Allocated bitmap for 4088 MSIs (base IRQ 0xfe000)
[    0.000000] Initializing IODA2 PHB (/pciex@600c3c0100000)
[    0.000000] PCI host bridge /pciex@600c3c0100000  ranges:
[    0.000000]  MEM 0x000600c080000000..0x000600c0fffeffff -> 0x0000000080000000 
[    0.000000] ioremap() called early from pnv_pci_init_ioda_phb+0x388/0xc90. Use early_ioremap() instead
[    0.000000]  MEM 0x0006004000000000..0x0006007fffffffff -> 0x0006004000000000 (M64 #1..15)
[    0.000000]  Using M64 #15 as default window
[    0.000000]   256 (255) PE's M32: 0x80000000 [segment=0x800000]
[    0.000000]                  M64: 0x4000000000 [segment=0x40000000]
[    0.000000]   Allocated bitmap for 2040 MSIs (base IRQ 0xfd800)
[    0.000000] Initializing IODA2 PHB (/pciex@600c3c0200000)
[    0.000000] PCI host bridge /pciex@600c3c0200000  ranges:
[    0.000000]  MEM 0x000600c100000000..0x000600c17ffeffff -> 0x0000000080000000 
[    0.000000] ioremap() called early from pnv_pci_init_ioda_phb+0x388/0xc90. Use early_ioremap() instead
[    0.000000]  MEM 0x0006008000000000..0x000600bfffffffff -> 0x0006008000000000 (M64 #1..15)
[    0.000000]  Using M64 #15 as default window
[    0.000000]   256 (255) PE's M32: 0x80000000 [segment=0x800000]
[    0.000000]                  M64: 0x4000000000 [segment=0x40000000]
[    0.000000]   Allocated bitmap for 2040 MSIs (base IRQ 0xfd000)
[    0.000000] Initializing IODA2 PHB (/pciex@600c3c0300000)
[    0.000000] PCI host bridge /pciex@600c3c0300000  ranges:
[    0.000000]  MEM 0x000600c180000000..0x000600c1fffeffff -> 0x0000000080000000 
[    0.000000] ioremap() called early from pnv_pci_init_ioda_phb+0x388/0xc90. Use early_ioremap() instead
[    0.000000]  MEM 0x0006020000000000..0x0006023fffffffff -> 0x0006020000000000 (M64 #1..31)
[    0.000000]  Using M64 #31 as default window
[    0.000000]   512 (511) PE's M32: 0x80000000 [segment=0x400000]
[    0.000000]                  M64: 0x4000000000 [segment=0x20000000]
[    0.000000]   Allocated bitmap for 4088 MSIs (base IRQ 0xfc000)
[    0.000000] Initializing IODA2 PHB (/pciex@600c3c0400000)
[    0.000000] PCI host bridge /pciex@600c3c0400000  ranges:
[    0.000000]  MEM 0x000600c200000000..0x000600c27ffeffff -> 0x0000000080000000 
[    0.000000] ioremap() called early from pnv_pci_init_ioda_phb+0x388/0xc90. Use early_ioremap() instead
[    0.000000]  MEM 0x0006024000000000..0x0006027fffffffff -> 0x0006024000000000 (M64 #1..15)
[    0.000000]  Using M64 #15 as default window
[    0.000000]   256 (255) PE's M32: 0x80000000 [segment=0x800000]
[    0.000000]                  M64: 0x4000000000 [segment=0x40000000]
[    0.000000]   Allocated bitmap for 2040 MSIs (base IRQ 0xfb800)
[    0.000000] Initializing IODA2 PHB (/pciex@600c3c0500000)
[    0.000000] PCI host bridge /pciex@600c3c0500000  ranges:
[    0.000000]  MEM 0x000600c280000000..0x000600c2fffeffff -> 0x0000000080000000 
[    0.000000] ioremap() called early from pnv_pci_init_ioda_phb+0x388/0xc90. Use early_ioremap() instead
[    0.000000]  MEM 0x0006028000000000..0x000602bfffffffff -> 0x0006028000000000 (M64 #1..15)
[    0.000000]  Using M64 #15 as default window
[    0.000000]   256 (255) PE's M32: 0x80000000 [segment=0x800000]
[    0.000000]                  M64: 0x4000000000 [segment=0x40000000]
[    0.000000]   Allocated bitmap for 2040 MSIs (base IRQ 0xfb000)
[    0.000000] OPAL nvram setup, 589824 bytes
[    0.000000] barrier-nospec: using ORI speculation barrier
[    0.000000] barrier-nospec: patched 454 locations
[    0.000000] Top of RAM: 0x1ff400000, Total RAM: 0x1ff300000
[    0.000000] Memory hole size: 1MB
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x00000001ff3fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x00000000ffdfffff]
[    0.000000]   node   0: [mem 0x00000000fff00000-0x00000001ff3fffff]
[    0.000000] Zeroed struct page in unavailable ranges: 16 pages
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ff3fffff]
[    0.000000] On node 0 totalpages: 130864
[    0.000000]   Normal zone: 112 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 130864 pages, LIFO batch:3
[    0.000000] percpu: Embedded 2 pages/cpu s76056 r0 d55016 u131072
[    0.000000] pcpu-alloc: s76056 r0 d55016 u131072 alloc=2*65536
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07 
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 130752
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=tty0 console=hvc0 powersave=off quiet
[    0.000000] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 8235776K/8375296K available (13760K kernel code, 1088K rwdata, 2752K rodata, 4736K init, 1999K bss, 139520K rese
rved, 0K cma-reserved)
[    0.000000] random: get_random_u64 called from __kmem_cache_create+0x38/0x600 with crng_init=0
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=16.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=16
[    0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[    0.000000] xive: Interrupt handling initialized with native backend
[    0.000000] xive: Using priority 7 for all interrupts
[    0.000000] xive: Using 64kB queues
[    0.000000] time_init: decrementer frequency = 512.000000 MHz
[    0.000000] time_init: processor frequency   = 18446744071914.584320 MHz
[    0.000004] time_init: 56 bit decrementer (max: 7fffffffffffff)
[    0.000008] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[    0.000010] clocksource: timebase mult[1f40000] shift[24] registered
[    0.000018] clockevent: decrementer mult[83126f] shift[24] cpu[0]
[    0.000076] Console: colour dummy device 80x25
[    0.000081] printk: console [tty0] enabled
[    0.000083] printk: console [hvc0] enabled
[    0.000085] printk: bootconsole [udbg0] disabled
[    0.000120] pid_max: default: 32768 minimum: 301
[    0.000184] LSM: Security Framework initializing
[    0.000219] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes, linear)
[    0.000231] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes, linear)
[    0.000819] EEH: PowerNV platform initialized
[    0.000823] POWER9 performance monitor hardware support registered
[    0.000854] rcu: Hierarchical SRCU implementation.
[    0.001076] smp: Bringing up secondary CPUs ...
[    0.005666] smp: Brought up 1 node, 16 CPUs
[    0.005668] numa: Node 0 CPUs: 0-15
[    0.005671] Using shared cache scheduler topology
[    0.006576] devtmpfs: initialized
[    0.013674] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.013680] futex hash table entries: 4096 (order: 3, 524288 bytes, linear)
[    0.013842] xor: measuring software checksum speed
[    0.110060]    8regs     :  5248.000 MB/sec
[    0.210092]    8regs_prefetch:  4243.200 MB/sec
[    0.310115]    32regs    :  5344.000 MB/sec
[    0.410139]    32regs_prefetch:  4166.400 MB/sec
[    0.510173]    altivec   :  6950.400 MB/sec
[    0.510175] xor: using function: altivec (6950.400 MB/sec)
[    0.510279] NET: Registered protocol family 16
[    0.510645] cpuidle: using governor menu
[    0.515185] PCI: Probing PCI hardware
[    0.515223] PCI host bridge to bus 0000:00
[    0.515230] pci_bus 0000:00: root bus resource [mem 0x600c000000000-0x600c07ffeffff] (bus address [0x80000000-0xfffeffff])
[    0.515234] pci_bus 0000:00: root bus resource [mem 0x6000000000000-0x6003fbfffffff 64bit pref]
[    0.515237] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.515241] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to ff
[    0.515276] pci 0000:00:00.0: [1014:04c1] type 01 class 0x060400
[    0.515499] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    0.521428] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.521468] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to ff
[    0.521500] PCI host bridge to bus 0001:00
[    0.521504] pci_bus 0001:00: root bus resource [mem 0x600c080000000-0x600c0fffeffff] (bus address [0x80000000-0xfffeffff])
[    0.521507] pci_bus 0001:00: root bus resource [mem 0x6004000000000-0x6007f7fffffff 64bit pref]
[    0.521510] pci_bus 0001:00: root bus resource [bus 00-ff]
[    0.521513] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to ff
[    0.521539] pci 0001:00:00.0: [1014:04c1] type 01 class 0x060400
[    0.521752] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    0.527663] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    0.527702] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to ff
[    0.527732] PCI host bridge to bus 0002:00
[    0.527736] pci_bus 0002:00: root bus resource [mem 0x600c100000000-0x600c17ffeffff] (bus address [0x80000000-0xfffeffff])
[    0.527739] pci_bus 0002:00: root bus resource [mem 0x6008000000000-0x600bf7fffffff 64bit pref]
[    0.527742] pci_bus 0002:00: root bus resource [bus 00-ff]
[    0.527745] pci_bus 0002:00: busn_res: [bus 00-ff] end is updated to ff
[    0.527772] pci 0002:00:00.0: [1014:04c1] type 01 class 0x060400
[    0.527984] pci 0002:00:00.0: PME# supported from D0 D3hot D3cold
[    0.533885] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    0.533924] pci_bus 0002:00: busn_res: [bus 00-ff] end is updated to ff
[    0.533954] PCI host bridge to bus 0003:00
[    0.533959] pci_bus 0003:00: root bus resource [mem 0x600c180000000-0x600c1fffeffff] (bus address [0x80000000-0xfffeffff])
[    0.533961] pci_bus 0003:00: root bus resource [mem 0x6020000000000-0x6023fbfffffff 64bit pref]
[    0.533964] pci_bus 0003:00: root bus resource [bus 00-ff]
[    0.533967] pci_bus 0003:00: busn_res: [bus 00-ff] end is updated to ff
[    0.533993] pci 0003:00:00.0: [1014:04c1] type 01 class 0x060400
[    0.534206] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    0.537268] pci 0003:01:00.0: [104c:8241] type 00 class 0x0c0330
[    0.537325] pci 0003:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    0.537350] pci 0003:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff 64bit]
[    0.537419] pci 0003:01:00.0: BAR2 [mem size 0x00002000 64bit]: requesting alignment to 0x10000
[    0.537539] pci 0003:01:00.0: supports D1 D2
[    0.537542] pci 0003:01:00.0: PME# supported from D0 D1 D2 D3hot
[    0.540563] pci 0003:00:00.0: PCI bridge to [bus 01]
[    0.540640] pci_bus 0003:00: busn_res: [bus 00-ff] end is updated to 01
[    0.540670] PCI host bridge to bus 0004:00
[    0.540674] pci_bus 0004:00: root bus resource [mem 0x600c200000000-0x600c27ffeffff] (bus address [0x80000000-0xfffeffff])
[    0.540676] pci_bus 0004:00: root bus resource [mem 0x6024000000000-0x6027f7fffffff 64bit pref]
[    0.540679] pci_bus 0004:00: root bus resource [bus 00-ff]
[    0.540682] pci_bus 0004:00: busn_res: [bus 00-ff] end is updated to ff
[    0.540708] pci 0004:00:00.0: [1014:04c1] type 01 class 0x060400
[    0.540921] pci 0004:00:00.0: PME# supported from D0 D3hot D3cold
[    0.544001] pci 0004:01:00.0: [14e4:1657] type 00 class 0x020000
[    0.544071] pci 0004:01:00.0: reg 0x10: [mem 0x6024000000000-0x602400000ffff 64bit pref]
[    0.544098] pci 0004:01:00.0: reg 0x18: [mem 0x6024000010000-0x602400001ffff 64bit pref]
[    0.544125] pci 0004:01:00.0: reg 0x20: [mem 0x6024000020000-0x602400002ffff 64bit pref]
[    0.544142] pci 0004:01:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[    0.544174] pci 0004:01:00.0: BAR6 [mem size 0x00000800 pref]: requesting alignment to 0x10000
[    0.544371] pci 0004:01:00.0: PME# supported from D0 D3hot D3cold
[    0.544433] pci 0004:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x2 link at 0004:00:00.0 (capable of 8.000 Gb/
s with 2.5 GT/s x4 link)
[    0.544528] pci 0004:01:00.1: [14e4:1657] type 00 class 0x020000
[    0.544597] pci 0004:01:00.1: reg 0x10: [mem 0x6024000030000-0x602400003ffff 64bit pref]
[    0.544625] pci 0004:01:00.1: reg 0x18: [mem 0x6024000040000-0x602400004ffff 64bit pref]
[    0.544652] pci 0004:01:00.1: reg 0x20: [mem 0x6024000050000-0x602400005ffff 64bit pref]
[    0.544669] pci 0004:01:00.1: reg 0x30: [mem 0x00000000-0x000007ff pref]
[    0.544701] pci 0004:01:00.1: BAR6 [mem size 0x00000800 pref]: requesting alignment to 0x10000
[    0.544899] pci 0004:01:00.1: PME# supported from D0 D3hot D3cold
[    0.547901] pci 0004:00:00.0: PCI bridge to [bus 01]
[    0.547996] pci_bus 0004:00: busn_res: [bus 00-ff] end is updated to 01
[    0.548026] PCI host bridge to bus 0005:00
[    0.548030] pci_bus 0005:00: root bus resource [mem 0x600c280000000-0x600c2fffeffff] (bus address [0x80000000-0xfffeffff])
[    0.548033] pci_bus 0005:00: root bus resource [mem 0x6028000000000-0x602bf7fffffff 64bit pref]
[    0.548036] pci_bus 0005:00: root bus resource [bus 00-ff]
[    0.548039] pci_bus 0005:00: busn_res: [bus 00-ff] end is updated to ff
[    0.548065] pci 0005:00:00.0: [1014:04c1] type 01 class 0x060400
[    0.548276] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    0.551342] pci 0005:01:00.0: [1a03:1150] type 01 class 0x060400
[    0.551472] pci 0005:01:00.0: enabling Extended Tags
[    0.551624] pci 0005:01:00.0: supports D1 D2
[    0.551626] pci 0005:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.554639] pci 0005:00:00.0: PCI bridge to [bus 01-02]
[    0.554768] pci_bus 0005:02: extended config space not accessible
[    0.554819] pci 0005:02:00.0: [1a03:2000] type 00 class 0x030000
[    0.554858] pci 0005:02:00.0: reg 0x10: [mem 0x600c280000000-0x600c280ffffff]
[    0.554878] pci 0005:02:00.0: reg 0x14: [mem 0x600c281000000-0x600c28101ffff]
[    0.554899] pci 0005:02:00.0: reg 0x18: [io  0x0000-0x007f]
[    0.555061] pci 0005:02:00.0: supports D1 D2
[    0.555064] pci 0005:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.558027] pci 0005:01:00.0: PCI bridge to [bus 02]
[    0.558092] pci_bus 0005:00: busn_res: [bus 00-ff] end is updated to 02
[    0.558108] pci 0000:00     : [PE# 1fe] Secondary bus 0x0000000000000000 associated with PE#1fe
[    0.558347] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.558369] pci_bus 0000:00: resource 4 [mem 0x600c000000000-0x600c07ffeffff]
[    0.558372] pci_bus 0000:00: resource 5 [mem 0x6000000000000-0x6003fbfffffff 64bit pref]
[    0.558377] pci 0001:00     : [PE# fe] Secondary bus 0x0000000000000000 associated with PE#fe
[    0.558612] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    0.558634] pci_bus 0001:00: resource 4 [mem 0x600c080000000-0x600c0fffeffff]
[    0.558636] pci_bus 0001:00: resource 5 [mem 0x6004000000000-0x6007f7fffffff 64bit pref]
[    0.558640] pci 0002:00     : [PE# fe] Secondary bus 0x0000000000000000 associated with PE#fe
[    0.558876] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    0.558897] pci_bus 0002:00: resource 4 [mem 0x600c100000000-0x600c17ffeffff]
[    0.558900] pci_bus 0002:00: resource 5 [mem 0x6008000000000-0x600bf7fffffff 64bit pref]
[    0.558907] pci 0003:00:00.0: BAR 8: assigned [mem 0x600c180000000-0x600c1803fffff]
[    0.558912] pci 0003:01:00.0: BAR 0: assigned [mem 0x600c180000000-0x600c18000ffff 64bit]
[    0.558933] pci 0003:01:00.0: BAR 2: assigned [mem 0x600c180010000-0x600c180011fff 64bit]
[    0.558954] pci 0003:00     : [PE# 1fe] Secondary bus 0x0000000000000000 associated with PE#1fe
[    0.559194] pci 0003:01     : [PE# 1fd] Secondary bus 0x0000000000000001 associated with PE#1fd
[    0.559434] pci 0003:01     : [PE# 1fd] Setting up 32-bit TCE table at 0..80000000
[    0.560440] IOMMU table initialized, virtual merging enabled
[    0.560444] pci 0003:01     : [PE# 1fd] Setting up window#0 0..ffffffff pg=10000
[    0.560451] pci 0003:01     : [PE# 1fd] Enabling 64-bit DMA bypass
[    0.560455] pci 0003:00:00.0: PCI bridge to [bus 01]
[    0.560464] pci 0003:00:00.0:   bridge window [mem 0x600c180000000-0x600c1ffefffff]
[    0.560479] pci_bus 0003:00: resource 4 [mem 0x600c180000000-0x600c1fffeffff]
[    0.560482] pci_bus 0003:00: resource 5 [mem 0x6020000000000-0x6023fbfffffff 64bit pref]
[    0.560484] pci_bus 0003:01: resource 1 [mem 0x600c180000000-0x600c1ffefffff]
[    0.560490] pci 0004:00:00.0: BAR 9: assigned [mem 0x6024000000000-0x602403fffffff 64bit pref]
[    0.560493] pci 0004:00:00.0: BAR 8: assigned [mem 0x600c200000000-0x600c2007fffff]
[    0.560498] pci 0004:01:00.0: BAR 0: assigned [mem 0x6024000000000-0x602400000ffff 64bit pref]
[    0.560520] pci 0004:01:00.0: BAR 2: assigned [mem 0x6024000010000-0x602400001ffff 64bit pref]
[    0.560542] pci 0004:01:00.0: BAR 4: assigned [mem 0x6024000020000-0x602400002ffff 64bit pref]
[    0.560563] pci 0004:01:00.0: BAR 6: assigned [mem 0x600c200000000-0x600c2000007ff pref]
[    0.560567] pci 0004:01:00.1: BAR 0: assigned [mem 0x6024000030000-0x602400003ffff 64bit pref]
[    0.560588] pci 0004:01:00.1: BAR 2: assigned [mem 0x6024000040000-0x602400004ffff 64bit pref]
[    0.560610] pci 0004:01:00.1: BAR 4: assigned [mem 0x6024000050000-0x602400005ffff 64bit pref]
[    0.560632] pci 0004:01:00.1: BAR 6: assigned [mem 0x600c200010000-0x600c2000107ff pref]
[    0.560636] pci 0004:00     : [PE# fe] Secondary bus 0x0000000000000000 associated with PE#fe
[    0.560886] pci 0004:01     : [PE# 00] Secondary bus 0x0000000000000001 associated with PE#0
[    0.561124] pci 0004:01     : [PE# 00] Setting up 32-bit TCE table at 0..80000000
[    0.562126] pci 0004:01     : [PE# 00] Setting up window#0 0..ffffffff pg=10000
[    0.562131] pci 0004:01     : [PE# 00] Enabling 64-bit DMA bypass
[    0.562135] pci 0004:00:00.0: PCI bridge to [bus 01]
[    0.562144] pci 0004:00:00.0:   bridge window [mem 0x600c200000000-0x600c27fefffff]
[    0.562150] pci 0004:00:00.0:   bridge window [mem 0x6024000000000-0x6027f7ff0ffff 64bit pref]
[    0.562161] pci_bus 0004:00: resource 4 [mem 0x600c200000000-0x600c27ffeffff]
[    0.562164] pci_bus 0004:00: resource 5 [mem 0x6024000000000-0x6027f7fffffff 64bit pref]
[    0.562167] pci_bus 0004:01: resource 1 [mem 0x600c200000000-0x600c27fefffff]
[    0.562170] pci_bus 0004:01: resource 2 [mem 0x6024000000000-0x6027f7ff0ffff 64bit pref]
[    0.562176] pci 0005:00:00.0: BAR 8: assigned [mem 0x600c280000000-0x600c2817fffff]
[    0.562179] pci 0005:00:00.0: BAR 7: no space for [io  size 0x1000]
[    0.562181] pci 0005:00:00.0: BAR 7: failed to assign [io  size 0x1000]
[    0.562186] pci 0005:01:00.0: BAR 8: assigned [mem 0x600c280000000-0x600c2817fffff]
[    0.562188] pci 0005:01:00.0: BAR 7: no space for [io  size 0x1000]
[    0.562191] pci 0005:01:00.0: BAR 7: failed to assign [io  size 0x1000]
[    0.562195] pci 0005:02:00.0: BAR 0: assigned [mem 0x600c280000000-0x600c280ffffff]
[    0.562203] pci 0005:02:00.0: BAR 1: assigned [mem 0x600c281000000-0x600c28101ffff]
[    0.562211] pci 0005:02:00.0: BAR 2: no space for [io  size 0x0080]
[    0.562214] pci 0005:02:00.0: BAR 2: failed to assign [io  size 0x0080]
[    0.562217] pci 0005:00     : [PE# fe] Secondary bus 0x0000000000000000 associated with PE#fe
[    0.562457] pci 0005:02     : [PE# fd] Secondary bus 0x0000000000000002..0x0000000000000002 associated with PE#fd
[    0.562695] pci 0005:02     : [PE# fd] Setting up 32-bit TCE table at 0..80000000
[    0.563700] pci 0005:02     : [PE# fd] Setting up window#0 0..ffffffff pg=10000
[    0.563706] pci 0005:02     : [PE# fd] Enabling 64-bit DMA bypass
[    0.563709] pci 0005:01:00.0: PCI bridge to [bus 02]
[    0.563719] pci 0005:01:00.0:   bridge window [mem 0x600c280000000-0x600c2ffefffff]
[    0.563737] pci 0005:01     : [PE# fc] Secondary bus 0x0000000000000001 associated with PE#fc
[    0.563976] pci 0005:00:00.0: PCI bridge to [bus 01-02]
[    0.563985] pci 0005:00:00.0:   bridge window [mem 0x600c280000000-0x600c2ffefffff]
[    0.563999] pci_bus 0005:00: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.564002] pci_bus 0005:00: resource 4 [mem 0x600c280000000-0x600c2fffeffff]
[    0.564005] pci_bus 0005:00: resource 5 [mem 0x6028000000000-0x602bf7fffffff 64bit pref]
[    0.564007] pci_bus 0005:01: resource 1 [mem 0x600c280000000-0x600c2ffefffff]
[    0.564010] pci_bus 0005:02: resource 1 [mem 0x600c280000000-0x600c2ffefffff]
[    0.564684] pci 0003:01:00.0: Adding to iommu group 0
[    0.564697] pci 0004:01:00.0: Adding to iommu group 1
[    0.564702] pci 0004:01:00.1: Adding to iommu group 1
[    0.564715] pci 0005:02:00.0: Adding to iommu group 2
[    0.564751] pci 0003:00:00.0: enabling device (0105 -> 0107)
[    0.564766] pci 0004:00:00.0: enabling device (0105 -> 0107)
[    0.564780] pci 0005:00:00.0: enabling device (0105 -> 0107)
[    0.564795] pci 0005:01:00.0: enabling device (0145 -> 0147)
[    0.564807] EEH: Capable adapter found: recovery enabled.
[    0.564875] PCI: Probing PCI hardware done
[    0.740244] raid6: vpermxor8 gen()  5102 MB/s
[    0.910284] raid6: vpermxor4 gen()  5162 MB/s
[    1.080334] raid6: vpermxor2 gen()  4428 MB/s
[    1.250397] raid6: vpermxor1 gen()  3068 MB/s
[    1.420436] raid6: altivecx8 gen()  4414 MB/s
[    1.590472] raid6: altivecx4 gen()  5041 MB/s
[    1.760528] raid6: altivecx2 gen()  4428 MB/s
[    1.930577] raid6: altivecx1 gen()  3070 MB/s
[    2.100628] raid6: int64x8  gen()  2258 MB/s
[    2.270672] raid6: int64x8  xor()   834 MB/s
[    2.440716] raid6: int64x4  gen()  3048 MB/s
[    2.610756] raid6: int64x4  xor()   974 MB/s
[    2.780819] raid6: int64x2  gen()  2693 MB/s
[    2.950882] raid6: int64x2  xor()   825 MB/s
[    3.120921] raid6: int64x1  gen()  1684 MB/s
[    3.290979] raid6: int64x1  xor()   526 MB/s
[    3.290981] raid6: using algorithm vpermxor4 gen() 5162 MB/s
[    3.290982] raid6: using intx1 recovery algorithm
[    3.291078] iommu: Default domain type: Translated 
[    3.291165] pci 0005:02:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    3.291168] pci 0005:02:00.0: vgaarb: bridge control possible
[    3.291172] pci 0005:02:00.0: vgaarb: setting as boot device (VGA legacy resources not available)
[    3.291174] vgaarb: loaded
[    3.291325] SCSI subsystem initialized
[    3.291388] libata version 3.00 loaded.
[    3.291421] usbcore: registered new interface driver usbfs
[    3.291436] usbcore: registered new interface driver hub
[    3.291485] usbcore: registered new device driver usb
[    3.291742] clocksource: Switched to clocksource timebase
[    3.291883] VFS: Disk quotas dquot_6.6.0
[    3.291933] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    3.293755] NET: Registered protocol family 2
[    3.293925] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes, linear)
[    3.293952] TCP established hash table entries: 65536 (order: 3, 524288 bytes, linear)
[    3.294204] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[    3.294487] TCP: Hash tables configured (established 65536 bind 65536)
[    3.294536] UDP hash table entries: 4096 (order: 1, 131072 bytes, linear)
[    3.294582] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes, linear)
[    3.294685] NET: Registered protocol family 1
[    3.294820] RPC: Registered named UNIX socket transport module.
[    3.294822] RPC: Registered udp transport module.
[    3.294823] RPC: Registered tcp transport module.
[    3.294824] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.294856] pci 0003:01:00.0: enabling device (0140 -> 0142)
[    3.294911] PCI: CLS 0 bytes, default 128
[    4.222869] Initialise system trusted keyrings
[    4.222934] workingset: timestamp_bits=54 max_order=17 bucket_order=0
[    4.225857] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    4.226138] NFS: Registering the id_resolver key type
[    4.226143] Key type id_resolver registered
[    4.226145] Key type id_legacy registered
[    4.226150] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    4.226160] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    4.226328] JFS: nTxBlock = 4021, nTxLock = 32171
[    4.227396] SGI XFS with ACLs, security attributes, realtime, quota, fatal assert, debug enabled
[    4.227934] F2FS not supported on PAGE_SIZE(65536) != 4096
[    4.228011] Key type asymmetric registered
[    4.228013] Asymmetric key parser 'x509' registered
[    4.228025] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    4.228376] IPMI message handler: version 39.2
[    4.228390] ipmi device interface
[    4.243364] ipmi-powernv ibm,opal:ipmi: IPMI message handler: The GUID response from the BMC was too short, it was 1 but should have
 been 17.  Assuming GUID is not available.
[    4.258499] ipmi-powernv ibm,opal:ipmi: IPMI message handler: Found new BMC (man_id: 0x00cb06, prod_id: 0x0001, dev_id: 0x00)
[    4.372809] hvc0: raw protocol on /ibm,opal/consoles/serial@0 (boot console)
[    4.372813] hvc0: No interrupts property, using OPAL event
[    4.373071] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    4.373342] Non-volatile memory driver v1.3
[    4.373419] [drm] radeon kernel modesetting enabled.
[    4.373519] ast 0005:02:00.0: enabling device (0141 -> 0143)
[    4.373605] [drm] platform has no IO space, trying MMIO
[    4.373608] [drm] Using device-tree for configuration
[    4.373610] [drm] AST 2500 detected
[    4.373613] [drm] Analog VGA only
[    4.373620] [drm] dram MCLK=800 Mhz type=7 bus_width=16 size=01000000
[    4.373672] [TTM] Zone  kernel: Available graphics memory: 4117888 KiB
[    4.373673] [TTM] Zone   dma32: Available graphics memory: 2097152 KiB
[    4.373675] [TTM] Initializing pool allocator
[    4.500810] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[    4.500813] mm/pgtable-generic.c:50: bad pmd c00600c2802001be.
[    4.500814] mm/pgtable-generic.c:50: bad pmd c00600c2804001be.
[    4.502335] Console: switching to colour frame buffer device 240x67
[    4.515173] ast 0005:02:00.0: fb0: astdrmfb frame buffer device
[    4.549620] [drm] Initialized ast 0.1.0 20120228 for 0005:02:00.0 on minor 0
[    4.552019] brd: module loaded
[    4.555379] loop: module loaded
[    4.555397] st: Version 20160209, fixed bufsize 32768, s/g segs 256
[    4.555834] 5 fixed-partitions partitions found on MTD device flash@0
[    4.555836] Creating 5 MTD partitions on "flash@0":
[    4.555840] 0x000000000000-0x000004000000 : "PNOR"
[    4.556367] 0x0000022a1000-0x000002e2a158 : "BOOTKERNEL"
[    4.556865] 0x000003944000-0x000003946000 : "VERSION"
[    4.557357] 0x000003989000-0x0000039c9000 : "IMA_CATALOG"
[    4.557855] 0x000003e10000-0x000003ff0000 : "BOOTKERNFW"
[    4.558465] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.558469] ohci-pci: OHCI PCI platform driver
[    4.581788] rtc-opal opal-rtc: registered as rtc0
[    4.581806] i2c /dev entries driver
[    4.584052] powernv-cpufreq: cpufreq pstate min 0x76 nominal 0x4e max 0x0
[    4.584054] powernv-cpufreq: Workload Optimized Frequency is enabled in the platform
[    4.584415] powernv-cpufreq: Frequency Control disabled from OS
[    4.584416] powernv-cpufreq: Frequency Control disabled from OS
[    4.584418] powernv-cpufreq: PMSR = 4e4e763080000000
[    4.584421] powernv-cpufreq: PMSR = 4e4e763080000000
[    4.584421] powernv-cpufreq: CPU Frequency could be throttled
[    4.584423] powernv-cpufreq: CPU Frequency could be throttled
[    4.584498] ipip: IPv4 and MPLS over IPv4 tunneling driver
[    4.584665] NET: Registered protocol family 17
[    4.584724] NET: Registered protocol family 33
[    4.584726] Key type rxrpc registered
[    4.584728] Key type rxrpc_s registered
[    4.584743] Key type dns_resolver registered
[    4.584792] drmem: No dynamic reconfiguration memory found
[    4.585110] registered taskstats version 1
[    4.585112] Loading compiled-in X.509 certificates
[    4.586989] Loaded X.509 cert 'Build time autogenerated kernel key: 39da3e2d7f2af95360454e5570b40bf8a6a3cbe5'
[    4.586996] kAFS: Red Hat AFS client v0.1 registering.
[    4.811750] kAFS: failed to register: -97
[    4.812443] Btrfs loaded, crc32c=crc32c-generic
[    4.812757] Key type encrypted registered
[    4.812783] ima: No TPM chip found, activating TPM-bypass!
[    4.812786] ima: Allocated hash algorithm: sha1
[    4.812799] ima: No architecture policies found
[    4.812816] evm: Initialising EVM extended attributes:
[    4.812817] evm: security.capability
[    4.812819] evm: HMAC attrs: 0x1
[    4.841770] rtc-opal opal-rtc: setting system clock to 2022-10-14T18:34:57 UTC (1665772497)
[    4.842626] Freeing unused kernel memory: 4736K
[    4.901780] Run /init as init process
[    4.903313] [U] hello world
[    5.140926] random: fast init done
[    7.507029] random: crng init done
[   90.053715] ehci_hcd: module verification failed: signature and/or required key missing - tainting kernel
[   90.053978] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   90.053979] Warning! ehci_hcd should always be loaded before uhci_hcd and ohci_hcd, not after
[   90.056346] ehci-pci: EHCI PCI platform driver
[   90.063445] xhci_hcd 0003:01:00.0: xHCI Host Controller
[   90.063517] xhci_hcd 0003:01:00.0: new USB bus registered, assigned bus number 1
[   90.063643] xhci_hcd 0003:01:00.0: hcc params 0x0270f06d hci version 0x96 quirks 0x0000000004000000
[   90.064266] hub 1-0:1.0: USB hub found
[   90.064281] hub 1-0:1.0: 4 ports detected
[   90.064447] xhci_hcd 0003:01:00.0: xHCI Host Controller
[   90.064486] xhci_hcd 0003:01:00.0: new USB bus registered, assigned bus number 2
[   90.064490] xhci_hcd 0003:01:00.0: Host supports USB 3.0 SuperSpeed
[   90.064525] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[   90.064683] hub 2-0:1.0: USB hub found
[   90.064696] hub 2-0:1.0: 4 ports detected
[   90.431750] usb 1-2: new high-speed USB device number 2 using xhci_hcd
[   90.611756] usb 1-3: new high-speed USB device number 3 using xhci_hcd
[   90.656807] hub 1-3:1.0: USB hub found
[   90.657155] hub 1-3:1.0: 5 ports detected
[   90.791930] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[   90.971751] usb 1-4: new high-speed USB device number 4 using xhci_hcd
[   91.017143] hub 1-4:1.0: USB hub found
[   91.017743] hub 1-4:1.0: 4 ports detected
[   91.101751] usb 1-3.1: new high-speed USB device number 5 using xhci_hcd
[   91.351754] usb 1-4.3: new high-speed USB device number 6 using xhci_hcd
[   92.069908] input: OpenBMC virtual_input as /devices/pci0003:00/0003:00:00.0/0003:01:00.0/usb1/1-3/1-3.1/1-3.1:1.0/0003:1D6B:0104.00
01/input/input0
[   92.131797] hid-generic 0003:1D6B:0104.0001: input: USB HID v1.01 Keyboard [OpenBMC virtual_input] on usb-0003:01:00.0-3.1/input0
[   92.133234] input: OpenBMC virtual_input as /devices/pci0003:00/0003:00:00.0/0003:01:00.0/usb1/1-3/1-3.1/1-3.1:1.1/0003:1D6B:0104.00
02/input/input1
[   92.133273] hid-generic 0003:1D6B:0104.0002: input: USB HID v1.01 Mouse [OpenBMC virtual_input] on usb-0003:01:00.0-3.1/input1
[   92.135697] hid-generic 0003:20A0:4109.0003: hiddev96: USB HID v1.10 Device [Nitrokey Nitrokey Storage] on usb-0003:01:00.0-4.3/inpu
t2
[   92.135728] usbcore: registered new interface driver usbhid
[   92.135729] usbhid: USB HID core driver
[  455.624279] usb-storage 1-2:1.0: USB Mass Storage device detected
[  455.624398] scsi host0: usb-storage 1-2:1.0
[  455.624558] usb-storage 2-1:1.0: USB Mass Storage device detected
[  455.624665] scsi host1: usb-storage 2-1:1.0
[  455.624766] usb-storage 1-4.3:1.0: USB Mass Storage device detected
[  455.624855] scsi host2: usb-storage 1-4.3:1.0
[  455.624941] usbcore: registered new interface driver usb-storage
[  456.632676] scsi 1:0:0:0: Direct-Access     Generic  STORAGE DEVICE   1532 PQ: 0 ANSI: 6
[  456.632889] sd 1:0:0:0: Attached scsi generic sg0 type 0
[  456.634979] scsi 2:0:0:0: Direct-Access     Nitrokey Nitrokey Storage 1.00 PQ: 0 ANSI: 0
[  456.637092] scsi 0:0:0:0: Direct-Access     Fanxiang  S101/240GB           PQ: 0 ANSI: 2
[  456.637287] sd 0:0:0:0: Attached scsi generic sg1 type 0
[  456.637479] scsi 2:0:0:1: Direct-Access     Nitrokey Nitrokey Storage 1.00 PQ: 0 ANSI: 0
[  456.637655] sd 2:0:0:0: Attached scsi generic sg2 type 0
[  456.637777] sd 2:0:0:1: Attached scsi generic sg3 type 0
[  456.637831] sd 0:0:0:0: [sdb] 468862128 512-byte logical blocks: (240 GB/224 GiB)
[  456.639727] sd 0:0:0:0: [sdb] Write Protect is off
[  456.639731] sd 0:0:0:0: [sdb] Mode Sense: 38 00 00 00
[  456.640081] sd 2:0:0:0: [sdc] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
[  456.641695] sd 0:0:0:0: [sdb] No Caching mode page found
[  456.641785] sd 0:0:0:0: [sdb] Assuming drive cache: write through
[  456.643513] sd 2:0:0:0: [sdc] Write Protect is on
[  456.643517] sd 2:0:0:0: [sdc] Mode Sense: 1b 00 80 00
[  456.646073] sd 2:0:0:1: [sdd] Attached SCSI removable disk
[  456.649556] sd 2:0:0:0: [sdc] No Caching mode page found
[  456.649630] sd 2:0:0:0: [sdc] Assuming drive cache: write through
[  456.654838]  sdb: sdb1 sdb2 sdb3
[  456.659823] sd 0:0:0:0: [sdb] Attached SCSI disk
[  456.668288]  sdc: sdc1
[  456.673661] sd 2:0:0:0: [sdc] Attached SCSI removable disk
[  456.932624] sd 1:0:0:0: [sda] 124735488 512-byte logical blocks: (63.9 GB/59.5 GiB)
[  456.932981] sd 1:0:0:0: [sda] Write Protect is off
[  456.932984] sd 1:0:0:0: [sda] Mode Sense: 21 00 00 00
[  456.933340] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  456.935906]  sda: sda1
[  456.937203] sd 1:0:0:0: [sda] Attached SCSI removable disk
[  475.163908] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[  552.714193] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
~ # 

# We're running before /tmp/config exists, but flash.sh needs it, so generate its initial version
combine_configs

echo "Reading ROM..."
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SergiiDmytruk That won't fly.

Even if flashrom was meant to read only once to /tmp (major TOCTOU for everything else that relies on that backup later on) this would mean adding ~30 seconds at each boot?!


~ # time flashrom -p linux_mtd -r /tmp/tst
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
real	0m 26.32s
user	0m 0.08s
sys	0m 25.91s

Since flashtools is already modified, why cbfs cannot be modified to be able to search for base and not rely on flashrom and then pnor to get cbfs content?

Copy link
Collaborator

@tlaurion tlaurion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, scripts read ROM 3 times and reading 64 MiB from flash 3 times takes noticeable amount of time.

Adding 30 seconds at each boot (if flashrom calls under flash.sh were replaced to be called only once for reading backup instead of the current 3x which is currently a bit less than 90 seconds now) cannot happen at each boot. 30 seconds to backup rom, add content in cbfs section and then flash back would be acceptable. But extracting cbfs at boot should not take 30 seconds.

Can't cbfs be modified under flashtools to find/interpret content through pnor, combining the functions?

@SergiiDmytruk what are the alternative approaches ?

@SergiiDmytruk
Copy link
Contributor Author

I forgot to mention that flash.sh reports an error. I don't think it's flashrom which fails, but that progress measurement function does (flashrom_progress). I haven't looked closely, but maybe output of linux_mtd programmer doesn't match the expected format in some way (at least the progress part, VERIFIED seems to be there).

Can't cbfs be modified under flashtools to find/interpret content through pnor, combining the functions?

See comments at the top, reading flash through /dev/mem is unlikely to work. I can try, but I wouldn't hope for it.

what are the alternative approaches ?

Well, I was also thinking of abusing fmap of flashrom to read PNOR header, parse it and then generate another fmap file to read only HBI section. It's somewhat ugly, but should be much faster.

No, wait. We can probably just use /dev/mtd0 as flashrom does. It uses fread() so maybe mmap() won't work, but that's just an inconvenience.

@SergiiDmytruk
Copy link
Contributor Author

Reading PNOR using /dev/mtd works:

~ # time cbfs -l
cbfs_master_header
fallback/romstage
fallback/ramstage
config
revision
build_info
1-cpu.dtb
2-cpus.dtb
fallback/payload
heads/initrd/etc/config.user

header_pointer
real    0m 0.48s
user    0m 0.02s
sys     0m 0.46s

If this is functionally OK, I'll clean up commit history in flashtools.

@tlaurion
Copy link
Collaborator

tlaurion commented Oct 21, 2022

@SergiiDmytruk here is my patch against flash.sh to have good rom size detected (with progress bar), modifying logic to that reading the logs don't break prior of having VERIFIED. in current $IN read (breaking loop everywhere needed) and also stopping from reading ROM 3 times: reading once and verifying content through flashrom should be sufficient prior of flashing.

Reuse and adapt as needed. EDIT: added under #1230
Merged this PR and flash.sh changes to be tested under https://app.circleci.com/pipelines/github/tlaurion/heads/1223

diff --git a/initrd/bin/flash.sh b/initrd/bin/flash.sh
index d06589ad..e0d9535c 100755
--- a/initrd/bin/flash.sh
+++ b/initrd/bin/flash.sh
@@ -33,7 +33,7 @@ flashrom_progress() {
     while true ; do
         prev_prev_word=$prev_word
         prev_word=$IN
-        read -r -d' ' IN || break
+        read -r -d' ' IN
         if [ "$total_bytes" != "0" ]; then
             current=$(echo "$IN" | grep -E -o '0x[0-9a-f]+-0x[0-9a-f]+:.*' | grep -E -o "0x[0-9a-f]+" | tail -n 1)
             if [ "${current}" != "" ]; then
@@ -51,6 +51,11 @@ flashrom_progress() {
                     echo "Total flash size : $total_bytes bytes"
                 fi
             fi
+            if [ "$prev_word"  == "total_size:" ]; then
+                # Next is total size in bytes
+                total_bytes=$(echo "$IN" | grep -E -o '[0-9]+')
+                echo "Total flash size : $total_bytes bytes"
+            fi
         fi
         if [ "$percent" -gt 99 ]; then
             spin_idx=4
@@ -77,14 +82,19 @@ flashrom_progress() {
             fi
             if echo "$IN" | grep "identical" > /dev/null ; then
                 status="done"
-		        echo ""
+                echo ""
                 echo "The flash contents are identical to the image being flashed."
+                break
             fi
         fi
         if [ "$status" == "verifying" ]; then
             if echo "${IN}" | grep "VERIFIED." > /dev/null ; then
                 status="done"
                 echo "The flash contents were verified and the image was flashed correctly."
+                break
+            elif echo "${IN}" | grep "FAILED" > /dev/null ; then
+                echo 'Error while verifying flash content'
+                break
             fi
         fi
     done
@@ -101,16 +111,8 @@ flashrom_progress() {
 flash_rom() {
   ROM=$1
   if [ "$READ" -eq 1 ]; then
-    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}.1" \
-    || die "$ROM: Read failed"
-    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}.2" \
-    || die "$ROM: Read failed"
-    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}.3" \
-    || die "$ROM: Read failed"
-    if [ `sha256sum ${ROM}.[123] | cut -f1 -d ' ' | uniq | wc -l` -eq 1 ]; then
-      mv ${ROM}.1 $ROM
-      rm ${ROM}.[23]
-    else
+    flashrom $CONFIG_FLASHROM_OPTIONS -r "${ROM}"
+    if ! flashrom $CONFIG_FLASHROM_OPTIONS -v "${ROM}"; then
       die "$ROM: Read inconsistent"
     fi
   else

@tlaurion
Copy link
Collaborator

@SergiiDmytruk tested #1230 successfully, and https://app.circleci.com/pipelines/github/tlaurion/heads/1223 as well under #1230 (comment). Waiting for better implementation/review there but this works with flash.sh fix to be able to inject public key in firmware without flash.sh failing.

@tlaurion
Copy link
Collaborator

@SergiiDmytruk : It still seems that Heads internal firmware upgrades won't be possible, though, even if Heads was stitching the files together, and consequently, that no fwupd support will be possible in the future?

For example, from within Heads:

~ # flash.sh -r /tmp/backup.rom
Board talos-2_server detected, continuing...
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Verifying flash... VERIFIED.
~ # flash.sh  /tmp/backup.rom
Board talos-2_server detected, continuing...
af949ef53947dbb6940cba038c29166281bd0dfbcbc2f82383b381389d6288f7  /tmp/talos-2_server.rom
Failed to find FMAP in ROM file: /tmp/talos-2_server.rom
+++ Adding heads/initrd/.gnupg/pubring.kbx to /tmp/talos-2_server.rom
Failed to find FMAP in ROM file: /tmp/talos-2_server.rom
Failed to write cbfs file to new ROM file

@tlaurion
Copy link
Collaborator

@SergiiDmytruk
Question: flashrom mtd is different then flashrom -p internal in the sense that all content is being flashed each time, not just the differences?

~ # time flash.sh -r /tmp/backup.rom
Board talos-2_server detected, continuing...
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading flash... done.
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Verifying flash... VERIFIED.
real	0m 53.18s
user	0m 0.62s
sys	0m 51.84s
~ # time flash.sh -c /tmp/backup.rom
Board talos-2_server detected, continuing...
af949ef53947dbb6940cba038c29166281bd0dfbcbc2f82383b381389d6288f7  /tmp/talos-2_server.rom

Initializing internal Flash Programmer
Total flash size : 67108864 bytes
Reading old flash contents. Please wait...
Flashing: [##################################################] (100%)
The flash contents are identical to the image being flashed.

real	1m 49.86s
user	0m 43.12s
sys	1m 26.68s

@tlaurion
Copy link
Collaborator

It is not possible to use the Options->GPG options-> Replace gpg key option giving the same error:

gpg: keybox '//.gnupg/pubring.kbx' created
gpg: //.gnupg/trustdb.gpg: trustdb created
gpg: key E7B4A71658E36A93: public key "Insurgo Technologies Libres / Open Technologies <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2023-04-20
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
Board talos-2_server detected, continuing...
36a496f975443980cffda2a911f7c0df2f90558af4b6b720038cd7fc27ddbccd  /tmp/talos-2_server.rom
Failed to find FMAP in ROM file: /tmp/talos-2_server.rom
+++ Adding heads/initrd/.gnupg/pubring.kbx to /tmp/talos-2_server.rom
Failed to find FMAP in ROM file: /tmp/talos-2_server.rom
Failed to write cbfs file to new ROM file

Note also the repetitive additional mtd_size messages above.
Basically, adding key seems to work on a clean bmc flashed rom but it is impossible to update rom content afterward?

@tlaurion
Copy link
Collaborator

BMC

root@talos:/tmp/images/flash_sh_adapted# mboxctl --lpc-state
LPC Bus Maps: Flash Device
root@talos:/tmp/images/flash_sh_adapted# pflash -r /tmp/talos.pnor
Reading to "/tmp/talos.pnor" from 0x00000000..0x04000000 !
[==================================================] 100% ETA:0s     
root@talos:/tmp/images/flash_sh_adapted# pflash  -F ../../talos.pnor -f -P HBB -p *.bootblock && pflash  -F ../../talos.pnor -f -P HBI -p *.rom && pflash  -F ../../talos.pnor -f -P BOOTKERNEL -p zImage.bundled && mboxctl --backend file:/tmp/talos.pnor
About to program "heads-talos-2_server-v0.2.0-1268-gc0d1c9ad.bootblock" at 0x00205000..0x0020c002 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
About to program "heads-talos-2_server-v0.2.0-1268-gc0d1c9ad.rom" at 0x00425000..0x00666200 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
About to program "zImage.bundled" at 0x022a1000..0x02e2a158 !
Programming & Verifying...
[==================================================] 100%
Updating actual size in partition header...
SetBackend: Success
root@talos:/tmp/images/flash_sh_adapted# mboxctl --lpc-state
LPC Bus Maps: BMC Memory
root@talos:/tmp/images/flash_sh_adapted# obmcutil poweron && obmc-console-client

Heads:

!!!!! User requested recovery shell
!!!!! Starting recovery shell
~ # gpg-gui.sh # add public key to running ROM and reflash option
gpg: key E7B4A71658E36A93: 1 signature not checked due to a missing key
gpg: //.gnupg/trustdb.gpg: trustdb created
gpg: key E7B4A71658E36A93: public key "Insurgo Technologies Libres / Open Technologies <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: no ultimately trusted keys found
gpg: inserting ownertrust of 6
gpg: inserting ownertrust of 6
gpg: inserting ownertrust of 6
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   3  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 3u
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
Board talos-2_server detected, continuing...
228ac5314a748c879cff76fdde38e7ab11dea85172647e0e1e1fb08bb6020874  /tmp/talos-2_server.rom

Initializing internal Flash Programmer
Total flash size : 67108864 bytes
Reading old flash contents. Please wait...
Flashing: [##################################################] (100%)
Verifying flash contents. Please wait...
The flash contents were verified and the image was flashed correctly.

Reboot.

Heads again. Gpg Options-> List GPG keys in your keyring: OK
GPG Options -> Replace GPG key(s) in the current rom and reflash

gpg: keybox '//.gnupg/pubring.kbx' created
gpg: //.gnupg/trustdb.gpg: trustdb created
gpg: key E7B4A71658E36A93: public key "Insurgo Technologies Libres / Open Technologies <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2023-04-20
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
mtd_size = 0x4000000
PART
Board talos-2_server detected, continuing...
c6fe70bf81965b9738942e4609fc2b849b253fbc9de39d8be774e7b2e9f1a386  /tmp/talos-2_server.rom
Failed to find FMAP in ROM file: /tmp/talos-2_server.rom
+++ Adding heads/initrd/.gnupg/pubring.kbx to /tmp/talos-2_server.rom
Failed to find FMAP in ROM file: /tmp/talos-2_server.rom
Failed to write cbfs file to new ROM file

@tlaurion
Copy link
Collaborator

@SergiiDmytruk Also note that after key injection upon reboot, the following is showed on console:

Linux/PowerPC load: 
Finalizing device tree... flat tree at 0x20b88ce0
[  368.469666403,3] LPC[000]: Got SYNC no-response error. Error address reg: 0xd0010080
[  368.469678867,6] IPMI: dropping non severe PEL event
[    4.504374] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[    4.504376] mm/pgtable-generic.c:50: bad pmd c00600c2802001be.
[    4.504377] mm/pgtable-generic.c:50: bad pmd c00600c2804001be.
[    4.781756] kAFS: failed to register: -97

@tlaurion
Copy link
Collaborator

tlaurion commented Oct 24, 2022

@SergiiDmytruk Sending you 2 files under single tarball over matrix:

  • talos.orig is before first boot of talos with regions modified.
  • talos.gpg is after key injection from within heads and after poweroff from heads (key injection succeeded)

Impossible to use talos.gpg as mboxctl file backend to replace keys through Heads. Anything attempting to flash, keeping states would result in Failed to find FMAP in ROM and Failed to write cbfs file to new ROM file atop of https://app.circleci.com/pipelines/github/tlaurion/heads/1223/workflows/d6e6e2e8-9425-4ac8-bb16-489f08608bd6/jobs/11147/artifacts

@SergiiDmytruk
Copy link
Contributor Author

It still seems that Heads internal firmware upgrades won't be possible, though, even if Heads was stitching the files together, and consequently, that no fwupd support will be possible in the future?

Not sure about fwupd, it will need some handling of PNOR/ECC to work, but otherwise it should work as long as there are no bugs.

Question: flashrom mtd is different then flashrom -p internal in the sense that all content is being flashed each time, not just the differences?

I don't think using MTD should have such an effect.

Note also the repetitive additional mtd_size messages above.

That's debug messages I forgot to remove.

Also note that after key injection upon reboot, the following is showed on console:

Don't see anything unusual there except for bad pmd which I probably haven't seen in this part of the log.

Thanks, I'll see what might be going wrong on updating flash more than once.

@tlaurion
Copy link
Collaborator

tlaurion commented Oct 25, 2022

Question: flashrom mtd is different then flashrom -p internal in the sense that all content is being flashed each time, not just the differences?

I don't think using MTD should have such an effect.

~ # time flash.sh -c /tmp/backup.rom
Board talos-2_server detected, continuing...
af949ef53947dbb6940cba038c29166281bd0dfbcbc2f82383b381389d6288f7 /tmp/talos-2_server.rom

Initializing internal Flash Programmer
Total flash size : 67108864 bytes
Reading old flash contents. Please wait...
Flashing: [##################################################] (100%)
The flash contents are identical to the image being flashed.

real 1m 49.86s
user 0m 43.12s
sys 1m 26.68s

Flashing the same flash image content backed up before (-r for backup, -c flag for clean flash without persistence of settings in cbfs) took 1m49 here.

@SergiiDmytruk Everything leads into thinking that the whole flash gets overwritten, even if flashing exactly the same ROM image that was backuped previously, as seen in output (and why I provided that output), questioning mtd.

If each read and verify takes around 30 seconds each, flashing and verifying around 30 seconds each, then we are at around 1m49s adding the operations. Where if the flashing operation of the same ROM was to be applied only on blocks that changed, it would be expected that the operation be reduced to less than a minute and no flashing happening at all, since the write operation would not happen since the output from flashrom comparing backup and to be flashed image would see that the to be flashed image is identical. Then there is no explanation for the additional 30 seconds for the flashing operation.

Try in your tests. You will get the same results. There is a flash operation happening, even though flashrom reports that content is identical, after the fact, as opoosed to internal programmer.

Also, writing into cbfs being the result of a rom backup, cbfs injection of a keyring, trustdb and possiblity a config.user overlay is supposed to be a change of a couple of kilobytes. So if mtd was only applying the changes, as the internal programmer does, the flash operation itself should take maximum 2-3 seconds, while verifying should take another 30 seconds. But this is not what is observed, hence me asking if mtd writes each blocks, not only the differences. Bug or feature: I do not know.

@tlaurion
Copy link
Collaborator

tlaurion commented Oct 25, 2022

Also note that after key injection upon reboot, the following is showed on console:

Don't see anything unusual there except for bad pmd which I probably haven't seen in this part of the log.

First thing I witness : [ 4.781756] kAFS: failed to register: -97 which is why I provided output.
First time I see this error, which is why I provided output.

@tlaurion
Copy link
Collaborator

tlaurion commented Oct 25, 2022

It still seems that Heads internal firmware upgrades won't be possible, though, even if Heads was stitching the files together, and consequently, that no fwupd support will be possible in the future?

Not sure about fwupd, it will need some handling of PNOR/ECC to work, but otherwise it should work as long as there are no bugs.

Question here is not just about fwupd, but more about how users are supposed to update Heads without loosing their settings (keyring, trustdb, config.user overlay which is copied to new firmware image under Heads when persisting setting option is used under Heads).

As of today, we are testing and this is ok. But every other board under Heads produce a final ROM image that Heads configuration states FLASHROM_OPTIONS to properly handle regions to be flashed from a whole firmware image. For "Legacy boards" (ex: x230-flash+ x230 vs x230-maximized), the FLASHROM_OPTIONS clearly states that only the BIOS region (coreboot+linux payload) is to be written in IFD region, where maximized boards contain everything stitched together to be wholly flashed (with flashrom only applying block differences throug erase+write). As of today, this is done through BMC to flash a clean ROM, overwritting bootblock, hostboot and petitboot regions, leaving other regions untouched, without setting persistence for testing. But the desired outcome, as for fwupd prior work, is to drop a firmware under /boot where Heads as direct access to flash. Which means that the current use case is missing a final packed image (just like legacy roms, faking ifd, not containing ME nor GBE, and only specifying BIOS region to be flashed, giving BMC flashing equivalent: HBB HBI and BOOTKERNEL regions (pflash -F ../../talos.pnor -f -P HBB -p *.bootblock && pflash -F ../../talos.pnor -f -P HBI -p *.rom && pflash -F ../../talos.pnor -f -P BOOTKERNEL -p zImage.bundled)

Otherwise we are otherwise saying here that future fwupd in OS will be able to write directly to mtd (equivalent of passing iomem=relaxed to kexec'ed kernel at all time?) where past work and pending PR under Heads is for Heads to be able to deal with ROMs that landed under /boot, and there, the mechanics of settings persistence through cbfs is supposed to copy those settings into the ROM to be flashed (coreboot region inside of cbfs) to be able to flash the final combined result.

Either a properly packed Heads ROM + proper FLASHROM_OPTIONS are expected to instruct flashrom to flash specific mtd regions (partitions) with bootblock, coreboot and zImage.bundled from flashrom obtained backup (just like pflash is doing it from bmc), or as I understand it, Heads internal upgrade functionality (settings persistence of gpg keyring, trustdb and config.user overlay) will not be possible as for other boards?

@SergiiDmytruk
Copy link
Contributor Author

First thing I witness : [ 4.781756] kAFS: failed to register: -97 which is why I provided output.
First time I see this error, which is why I provided output.

That's the usual thing as far as I can tell. I see it all the time. I see the same:

[   24.075033188,3] LPC[000]: Got SYNC no-response error. Error address reg: 0xd0010080
[   24.075045102,6] IPMI: dropping non severe PEL event
[   24.075068057,7] UART: IRQ functional !
[    4.292746] mm/pgtable-generic.c:50: bad pmd c00600c2800001be.
[    4.571799] kAFS: failed to register: -97

Otherwise we are otherwise saying here that future fwupd in OS will be able to write directly to mtd (equivalent of passing iomem=relaxed to kexec'ed kernel at all time?)

MTD is not accessed through memory directly, kernel communicates with skiboot using OPAL API and asks it to read/write flash.

Either a properly packed Heads ROM + proper FLASHROM_OPTIONS are expected to instruct flashrom to flash specific mtd regions (partitions) with bootblock, coreboot and zImage.bundled from flashrom obtained backup (just like pflash is doing it from bmc), or as I understand it, Heads internal upgrade functionality (settings persistence of gpg keyring, trustdb and config.user overlay) will not be possible as for other boards?

flashrom isn't capable of dealing with PNOR on its own, either need to wrap it to read/write needed sections and post/pre-process them or write a separate tool that can do it. Part of this PR reads and writes via flashrom, but then exports/imports needed HBI partition to get to CBFS. Similar thing can be done for upgrade functionality, but it also requires updating actual size of PNOR partition (CBFS has fixed size, so size isn't updated by writes at the moment).

But this is not what is observed, hence me asking if mtd writes each blocks, not only the differences. Bug or feature: I do not know.

time flash.sh -c /tmp/backup.rom
Board talos-2_server detected, continuing...
5465f8d304ddb6b6b4c2fff0a06588447213660c116bd00dd249be6d56981c69  /tmp/talos-2_server.rom

Initializing internal Flash Programmer
Reading old flash contents. Please wait...
Flashing: [-                                                ] (0%)
The flash contents are identical to the image being flashed.

real    0m 51.35s
user    0m 11.20s
sys     0m 39.71s
time flashrom $CONFIG_FLASHROM_OPTIONS -w /tmp/backup.rom
flashrom  on Linux 5.5.0-openpower1 (ppc64le)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Opened /dev/mtd0 successfully
Found Programmer flash chip "Opaque flash chip" (65536 kB, Programmer-specific) on linux_mtd.
Reading old flash chip contents... done.
Erasing and writing flash chip...
Warning: Chip content is identical to the requested image.
Erase/write done.
real    0m 26.39s
user    0m 0.57s
sys     0m 25.47s

flash.sh takes twice more time than flashrom. Both are under 1m.

@SergiiDmytruk
Copy link
Contributor Author

Impossible to use talos.gpg as mboxctl file backend to replace keys through Heads. Anything attempting to flash, keeping states would result in Failed to find FMAP in ROM and Failed to write cbfs file to new ROM file atop of

Two more places needed cbfs.sh instead of cbfs. Seems to work with this change.

@tlaurion
Copy link
Collaborator

https://app.circleci.com/pipelines/github/tlaurion/heads?branch=SergiiDmytruk_flashrom_8a2cebf%2B5a7902c5ab6913a509ffff4db0a21f43d245305b includes your suggestions of #1230 (comment). Thanks, those are good improvements against master flash_progress code.

Hopefully that build will produce a valid zImage as opposed of last CircleCI build that needed to be built without using caches.

Will test when ready and report back.

~ # time flash.sh -c /tmp/backup 
Board talos-2_server detected, continuing...
2a7aea18d12a5fe38fa2c51918f218686f8d894eef8e2a988d2e69cf6d0604ab  /tmp/talos-2_server.rom

Initializing Flash Programmer
Total flash size : 67108864 bytes
Reading old flash contents. Please wait...
Flashing: [##################################################] (100%)
Verifying flash contents. Please wait...
The flash contents were verified and the image was flashed correctly.

real	1m 30.67s
user	0m 18.26s
sys	1m 41.92s
~ # time flash.sh -c /tmp/gpg-gui.rom 
Board talos-2_server detected, continuing...
9d94288318669b4d3072df904f39fdedabbc5b01de4966d4c28bcce8fe633c6b  /tmp/talos-2_server.rom

Initializing Flash Programmer
Total flash size : 67108864 bytes
Reading old flash contents. Please wait...
Flashing: [##################################################] (100%)
Verifying flash contents. Please wait...
The flash contents were verified and the image was flashed correctly.

real	1m 30.75s
user	0m 18.32s
sys	1m 42.71s
~ # sha256sum /bin/flash.sh 
70c37a61656f8d0b0592fa20836a351c325cc9724c57faaacf422de132710945  /bin/flash.sh

hashes from tested x230-htop-maximized hashes.txt file:
70c37a61656f8d0b0592fa20836a351c325cc9724c57faaacf422de132710945 ./bin/flash.sh

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 1, 2022

@SergiiDmytruk #1230 just got merged, please rebase on master.

@SergiiDmytruk
Copy link
Contributor Author

Rebased.

Heads should produce ROM images that are versioned correctly (bzImage.bundled not being an exception)

I think I didn't do this because zImage aren't versioned, but they probably aren't provided to the end user.

Heads should produce whole images aimed to be internally flashed from within Heads GUI and flash.sh

How about heads-talos-2_server-v0.X.0.tgz? And handle it specially in flash.sh.

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 2, 2022

Heads should produce ROM images that are versioned correctly (bzImage.bundled not being an exception)

I think I didn't do this because zImage aren't versioned, but they probably aren't provided to the end user.

The idea here is more related to consistent testing and knowing what is being deployed in case of Talos. For other boards, the images being self contained, it is not really an issue. But for Talos, 3 files are required, two of which are versioned and zImage.bundled is not. It is easy to report issues while testing a older zImage.bundled. They should all have the same file versioning scheme.

Heads should produce whole images aimed to be internally flashed from within Heads GUI and flash.sh

How about heads-talos-2_server-v0.X.0.tgz? And handle it specially in flash.sh.

Not against. If hashes.txt is being bundled in tarball, next steps could be to use the hashes.txt in the tarball to further verify firmware integrity as well. Similar changes exist in Nitrokey's fork, creating npf firmware upgrade packages and having flash.sh modified to deal with those here: master...Nitrokey:heads:nitropad. Could be upstreamed if that is useful to this usecase as well.

That is the easiest path vs padding those files into correctly constructed and interpretable flashrom images?

@SergiiDmytruk
Copy link
Contributor Author

That is the easiest path vs padding those files into correctly constructed and interpretable flashrom images?

Even if we assume that offsets of PNOR partitions are fixed and provide an appropriate fmap file/ifd flashrom still won't be able to update PNOR headers which might leave firmware in unbootable state.

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 2, 2022

Not against. If hashes.txt is being bundled in tarball, next steps could be to use the hashes.txt in the tarball to further verify firmware integrity as well. Similar changes exist in Nitrokey's fork, creating npf firmware upgrade packages and having flash.sh modified to deal with those here: master...Nitrokey:heads:nitropad. Could be upstreamed if that is useful to this usecase as well.

@SergiiDmytruk : then the above would be my recommendation. Maybe its time to provide compressed, versionned firmware packages for all boards if we go into that direction. I might generalize what will be done for Talos II to other boards as well.

Having the hashes.txt included in the tarball with localized hashes might be ideal to go forward as well, providing an easy way to validate hashes from within Heads against /boot dropped firmware package (see #613 ), and then that firmware package being part of detached signed digest would also be a nice step forward; permitting him to validate integrity of binaries/libraries/scripts against a known latest firmware package dropped under /boot, while also permitting reflashing the same firmware, expecting the sealed measurements to be exactly the same (TOTP/HOTP being still congruent) on next reboot, producing the same TPMTOTP/HOTP challenge. The flash.sh script could even check for expected checksum of images under hashes.txt prior of permitting flashing.

@macpijan
Copy link
Contributor

macpijan commented Nov 3, 2022

Just to remind that this MR originated from issue reporting lack of flashrom support: Dasharo/dasharo-issues#190

After some struggle we implemented some solution, even though from the start it was said that the internal programmer would not work for the POWER9.

Now we are discussing some versioning, even though currently no boards in heads implements that (?). I'm already confused what's left here to mark it as DONE.

I take this comment as the final needs.

#1230 needs to be merged, otherwise this PR as it is cannot be fully tested.

Looks merged already.

Heads should produce ROM images that are versioned correctly (bzImage.bundled not being an exception)

Looks like we agreed on how it can be done and @SergiiDmytruk will proceed?

Heads should produce whole images aimed to be internally flashed from within Heads GUI and flash.sh ...

I'm not sure where we are standing there

@SergiiDmytruk
Copy link
Contributor Author

Looks like we agreed on how it can be done and @SergiiDmytruk will proceed?

Sure.

Heads should produce whole images aimed to be internally flashed from within Heads GUI and flash.sh ...

I'm not sure where we are standing there

Packing 3 files into a tarball and doing things like for nitropad. Also need to update new pnor in flashtools a bit to make it update headers and add an option to not add ECC.

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 3, 2022

Now we are discussing some versioning, even though currently no boards in heads implements that (?).

@macpijan: if we look at artifacts currently produced by board configs under master https://app.circleci.com/pipelines/github/osresearch/heads/457/workflows/52255961-041d-41af-a1db-d73ff7772ffa/jobs/4964/artifacts

Talos boards currently produces BMC-only flashable artifacts:

My request here is to have consistent versioned filenames so that parts are clear to belong to the same board and same parent commit, as all other boards produced ROM artifacts.
What is missing here to ease testing/flashing by anyone and proper reporting is something consistent to test/report about.

Minimally:

  • heads-talos-2_server-v0.2.0-1267-gf848070.bootblock
  • heads-talos-2_server-v0.2.0-1267-gf848070.rom
  • heads-talos-2_server-v0.2.0-1267-gf848070.zImage.bundled

I was thinking of something like the following to match/ease understanding of dasharo flashing instructions and local filenames being easy to target for testing reports/reproducibility of issues:

  • heads-talos-2_server-v0.2.0-1267-gf848070.HBB.coreboot.bootblock
  • heads-talos-2_server-v0.2.0-1267-gf848070.HBI.coreboot.rom
  • heads-talos-2_server-v0.2.0-1267-gf848070.BOOTKERNEL.zImage.bundled

As a result, a user could easily upload a "heads-tests" directory containing firmware parts to be flashed through BMC, understanding what are HBB HBI and BOOTKERNEL files and quickly be able to tell from which commit they are flashing the parts from:

  • user@talos-tests:~$ rsync -ravczz --inplace --delete heads-tests root@talos:/tmp/images/
  • root@talos:/tmp/images/heads-tests# mboxctl --backend vpnor
  • root@talos:/tmp/images/heads-tests# pflash -r /tmp/talos.pnor
  • root@talos:/tmp/images/heads-tests# pflash -F ../../talos.pnor -f -P HBB -p *HBB* && pflash -F ../../talos.pnor -f -P HBI -p *HBI* && pflash -F ../../talos.pnor -f -P BOOTKERNEL -p *BOOTKERNEL* && mboxctl --backend file:/tmp/talos.pnor
  • root@talos:/tmp/images/heads-tests# obmcutil poweron && obmc-console-client

I'm already confused what's left here to mark it as DONE.
I take #1222 (comment) as the final needs.

#1230 needs to be merged, otherwise this PR as it is cannot be fully tested.
Looks merged already.

Correct. With #1230 merged, Heads doesn't prevent Talos to use flashrom successfully through flash.sh, as expected from this PR content without it merged. Otherwise, flashing failed after verification of flash content.

Heads should produce ROM images that are versioned correctly (bzImage.bundled not being an exception)
Looks like we agreed on how it can be done and @SergiiDmytruk will proceed?

It is agreed that coreboot+heads/hostboot+heads/coreboot+petitboot/hostboot+petitboot can be tested without flashing that initial testing from end users per https://docs.dasharo.com/variants/talos_2/installation-manual/#testing-firmware-images-without-flashing.

Heads support always required persistence of pubkey+trustdb+config injection/extraction through cbfs, which always required flashrom+cbfs integration, otherwise Heads is not Heads. With TPM, Heads measures and seals those measurements as part of TPMTOTP/HOTP. I'm still confused today on how Heads can be considered Heads without flashrom+cbfs. Without a TPM to measure, seal and remote attest firmware and having a setuped TPM released disk encryption key to boot OS, how a platform can be considered fully supported through Heads.

Going back to the part of this MR into that fulfilling Heads support: coreboot.rom, coreboot.bootblock and zImage need to be packed somehow together to be downloadable and flashable from within Heads Options->Flash/Update the BIOS, with persistence of user config settings transferrable through cbfs into the new image prior of flashing. The same applied to Options->GPG Options->Add GPG key to running BIOS and reflash, Options->GPG Options->Replace GPG key(s) to running BIOS and reflash. Again, as of today, I cannot test TPM functions until I can buy a module from 3mdeb, replicating what users would do to be able to use Heads on a Talos. But that will be other issues triggered by defining a default boot option and adding a TPM disk encryption key into the TPM, and having cryptsetup2 in board configs to be able to interact with LUKS headers and NVME PCI SSD drive, and out of scope for this MR as well. But the options are there under Heads to be tested directly through GUI. They should work in the scope of a specific MR and or course be functional prior of merging, without causing regression for other boards currently supported under Heads. Note that TPM is required to test Factory Reset/Re-Ownership functions as well, which will be scoped in other issues/MR later on when I have a TPM module to test as well. This code path also interacts with key injection and reflashing. But I cannot test it fully on my side, while you could.

@SergiiDmytruk : because there is no single Heads provided file nor board defined CONFIG_FLASHROM_OPTIONS that can actually take care of Heads upgrade? Not saying it is not feasible, I'm saying it is not currently possible.

Right, I didn't think of how those are delivered. An extra step could help: read flash and execute pnor rom --write HBB/HBI/BOOTKERNEL 3 times on it to produce firmware upgrade image, but pnor needs to update partition size on writes for this to work.

@SergiiDmytruk : If Heads/dasharo produced artifacts/documentation per board configuration are consistent to BMC memory/physical flashing/Heads internal flashing, then this PR can be merged and Dasharo/dasharo-issues#190 closed as complete. As of today, there is no internal upgrade possible from within Heads. So no real Heads support yet.

@SergiiDmytruk
Copy link
Contributor Author

Seems to work. Loaded server board, changed configuration, then successfully updated to workstation board retaining configuration.

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 7, 2022

  • versioned firmware filenames (rom, bootblock, zImage.bundled, tar.gz)
  • Internal flash of tgz without change: boots on reboot
  • GPG -> add public key, flash creates keyring and injects trustdb
    • Validate what are the 7x"Skipping..." messages on screen after injection of public key
  • Internal flash of same tar.gz firmware package keeping settings keeps persistence on reboot
    • Same 7x"Skipping..." messages
  • Test oem-factory-reset with USB Security dongle for the first time (Nitrokey Storage)
    • Same 7x"Skipping..." messages
    • gpg material is injected and persists on reboot
  • Setup boot default without TPM, hacking around calling usb-init from command line to have usb-strorage kernel modules loaded, so /dev/sdb2 is detected to boot from USB and detach sign /boot content. Reapply hack from command line, call gui-init and be able to boot default boot option with /boot being verified

@SergiiDmytruk
Copy link
Contributor Author

7x"Skipping..." messages

Logs again. Made the message more sensible and made it appear only with --verbose flag.

@tlaurion
Copy link
Collaborator

tlaurion commented Nov 9, 2022

Rebasing on master fixes server board's whiptail output that were not usable, as can be seen under #1237 and fixed under #1238,

Signed-off-by: Sergii Dmytruk <[email protected]>
This makes output suitable for use via Heads' menus.

Signed-off-by: Sergii Dmytruk <[email protected]>
@tlaurion
Copy link
Collaborator

@SergiiDmytruk par submitted to flashtools?

@tlaurion tlaurion merged commit 5e00800 into linuxboot:master Nov 11, 2022
@SergiiDmytruk SergiiDmytruk deleted the flashrom branch November 11, 2022 22:16
@SergiiDmytruk
Copy link
Contributor Author

I thought you'll approve this one, then I'll send flashtools PR to be able to update flashtools module before merging here :) Here it is: osresearch/flashtools#9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants