Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debugging kernel 5.x kexec issues add tools to ease kernel version bumping #1351

Commits on Mar 30, 2023

  1. WiP

    - config/linux-x230-[maximized,legacy,flash].config: real defconfig based on what was in tree without modifications
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    e396d9b View commit details
    Browse the repository at this point in the history
  2. WIP some more

    Added targets into modules/linux were used to generate oldconfigs for both 4.14.62 and 5.10.5, being then renamed to orig and mod along the way.
    Goal of all of this is to
    - Take defconfig format from 4.14.62 and remove unneeded compiled as modules but never packed (useless)
    - Investigate comparable configs between librem_common and xx30
    - Troubleshoot why in some case i915 not being provided in OS initrd is ok (xx30 on 4.14=ok vs t440p on 5.10 based on librem_common and librems)
      - That being outside coreboot iommu=on iommu_igfx=off passed from coreboot to Heads linux payload and with/without to OS
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    1542690 View commit details
    Browse the repository at this point in the history
  3. Wip continues

    As of now, only really important differences observed between librem and xx30 are
    - Some DRM addition for AST in librem and not required per xx30
    - NVME addition in librem and not required per xx30
    - Some additional processors support under librem that should not be there
    - Some additional sched support where I do not see anything usefull more then PERFORMANCE driven scheduler
    - SECURITY as base for SECURITYFS being activated but not being used
    - compression algos supported but unused
    - encryption compiled as modules but never packed
    - Hardware Randomization sources as TPM and Intel processor should be enabled to augment entropy and reduce delay before usable in kernel (more work needed there)
    
    This will be base for testing. Changes saved in oldconfoig format. Officially against defconfig format for github repo storage. Counterproductive.
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    6b55794 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6eea487 View commit details
    Browse the repository at this point in the history
  5. Wip continues

    kexec patch under tree takes into consideration that inteldrmfb hack added into kexec for 4.14 cannot apply anymore for 5.x
    Those changes changes linux-qemu.config to remove all unneeded, and take into account general advises to have vgaarb, fbdev and simple fb, as well as needed gpu (where qemu requires virtio related drivers)
    
    In all cases, kexec should be able to either be called reusing fb or resetting it.
    
    At this point, sure kexec patch won do it.
    That patch comes from older kexec, where nowadays it cannot be used as is.
    
    Some more tools added into modules/linux
    I mostly use:
    linux.modify_defconfig (to use defconfig, call menuconfig on it and save back in place)
    and
    linux.generate_and_save-oldconfig (to save whatever defconfig or modified oldconfig file into olconfig file where linux config is expected to be found per board config)
    
    Also kexec-boot was modified to add some additional kexec arguments in case of qemu usage, but nothing was successful, where traces are there to track what was attempted.
    At this point, going back to x230 to attempt to tweak vga options, reusing cleaning that was done on linux-qemu.conf to be reused there
    
    TODO:
    - Check advancements on kexec. Something better should be possible.
     - note that xen, multiboot and other usecases are differenciated in kexec-boot, where xen reuses vga.
       - other situation then xen doesn't pass any other kexec option to deal with vga, other then inteldrmfb patch in kexec as of today, which works for 4.14 kernel jumping into others newer, but doesn work for 5.x jumping to others as of today.
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    bf48c95 View commit details
    Browse the repository at this point in the history
  6. add i915drmfb to kexec patch, which match fb of 5.x+ kernels for inte…

    …l, but still not working
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    b184504 View commit details
    Browse the repository at this point in the history
  7. quick test reverting linux config to config/linux-x230-maximized.conf…

    …ig_oldconfig_5.10.5_from_orig_4.14.62
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    a43fa70 View commit details
    Browse the repository at this point in the history
  8. WiP VGA correctness

    Status
    qemu coreboot config:
     +CONFIG_GENERIC_LINEAR_FRAMEBUFFER=y momentarily
    
    qemu linux config:
    +CONFIG_X86_SYSFB=y seems to be requirement for non legacy VESA bringup, but seems to break Heads?!?
    removal of vga 16
    removal of vesa
    leaving simplefb
    
    NOTE: CONFIG_DRM_FBDEV_EMULATION is required and provides linux console support on top of modesetting driver
    
    modules/linux/coreboot: add modify helpers to generate in place oldconfigs and defconfigs
    
    WiP:
    kexec-boot: modifying kexec passed boot options on qemu for Tinycore.
    A bit confused on why kexec call just freezes there. Next time is investigating how to gdb into qemu for firmware, since we should have output on host console even if qemu is not able to get vga, and vga should be able to be reused from kexec call since:
    user@heads-tests:/media/boot$ zcat core.gz | cpio -ivt | grep -e vga -e fb -e drm -e virt
    lrwxrwxrwx   1 root     root           17 May  8  2022 usr/sbin/fbset -> ../../bin/busybox
    drwxr-xr-x   3 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/virt
    drwxr-xr-x   2 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/virt/lib
    -rw-r--r--   1 root     root         1779 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/virt/lib/irqbypass.ko.gz
    -rw-r--r--   1 root     root         4556 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/net/vmw_vsock/vmw_vsock_virtio_transport.ko.gz
    -rw-r--r--   1 root     root         6898 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko.gz
    drwxr-xr-x   3 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev
    -rw-r--r--   1 root     root         9612 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/udlfb.ko.gz
    drwxr-xr-x   2 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/core
    -rw-r--r--   1 root     root         2149 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/core/sysfillrect.ko.gz
    -rw-r--r--   1 root     root         1274 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/core/fb_sys_fops.ko.gz
    -rw-r--r--   1 root     root         1782 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/core/sysimgblt.ko.gz
    -rw-r--r--   1 root     root         2268 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/core/syscopyarea.ko.gz
    -rw-r--r--   1 root     root         7035 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/video/fbdev/hyperv_fb.ko.gz
    -rw-r--r--   1 root     root         3082 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/gpio/gpio-virtio.ko.gz
    drwxr-xr-x   3 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/drivers/virt
    drwxr-xr-x   2 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/drivers/virt/vboxguest
    -rw-r--r--   1 root     root        12506 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virt/vboxguest/vboxguest.ko.gz
    drwxr-xr-x   2 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/drivers/virtio
    -rw-r--r--   1 root     root         8029 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_ring.ko.gz
    -rw-r--r--   1 root     root         3847 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_pci_modern_dev.ko.gz
    -rw-r--r--   1 root     root         3718 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio.ko.gz
    -rw-r--r--   1 root     root         7084 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_pci.ko.gz
    -rw-r--r--   1 root     root         3377 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_input.ko.gz
    -rw-r--r--   1 root     root         6258 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_balloon.ko.gz
    -rw-r--r--   1 root     root         1042 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_dma_buf.ko.gz
    -rw-r--r--   1 root     root         3551 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/virtio/virtio_mmio.ko.gz
    -rw-r--r--   1 root     root         6916 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/block/virtio_blk.ko.gz
    -rw-r--r--   1 root     root        17961 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/net/virtio_net.ko.gz
    drwxr-xr-x   2 root     root            0 May  8  2022 lib/modules/5.15.10-tinycore/kernel/drivers/crypto/virtio
    -rw-r--r--   1 root     root         7020 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/crypto/virtio/virtio_crypto.ko.gz
    -rw-r--r--   1 root     root         5924 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/iommu/virtio-iommu.ko.gz
    -rw-r--r--   1 root     root        10007 Dec 21  2021 lib/modules/5.15.10-tinycore/kernel/drivers/char/virtio_console.ko.gz
    
    Last line is virtio_console that qemu is using.... So no clue at this point
    tlaurion committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    8872b12 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    6459416 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Finally some progress.

    Added debug capabilities inside of KERNEL_ADD in board config, also specifying ttys0 serial speed of 115200, which works now.
    Added debugging through kexec -d call as well if DEBUG output activated in board config
    
    Now Tinycore stalls at ethernet init, loop is established and resized multiple times, so it seems that tinycore has what he was searching for to prepare normal boot options in memory.
    Investigating vga options and will output some info on PR
    tlaurion committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    a39d71c View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2023

  1. Configuration menu
    Copy the full SHA
    b7ecf76 View commit details
    Browse the repository at this point in the history
  2. WiP: no changes really, still experiments

    qemu-coreboot: readd linear framebuffer
    qemu-linux:
    -add firmware options to try to help kexec
    -remove low and legacy fb options to only leave simple fb
    kexec-boot:
    -go back to only use --console-vga, but kexec -d debug is not showing anything interesting when booting tinycore
     - TINYCORE SHOULD BOOT WITH SIMPLE FB BUT DOESN'T
    modules/kexec: passed to latest (2.0.26)
    patches/kexec 2.0.22 patch copied over for 2.0.26 without changes
    tlaurion committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    dd59749 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9062bfe View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2023

  1. linux-qemu.config here is based on https://github.com/u-root/webboot/…

    …blob/78c6cf99a32d82b106cc373280df9512d6e25131/config-5.6.14, minus deactivating all uneeded network drivers but intel ones for Q35.
    
    This was accomplished by using added linux module build statement, which was called through:
    make BOARD=qemu-coreboot-whiptail-tpm1 linux.modify_and_save_oldconfig_in_place
    
    Changes of kexec-boot to pass 32 bit entry point + noefi is based on understanding of below comments.
    
    Keeping traces of testings traces
    
    --
    From linuxboot#1351 (comment)
    
    Knowledge on related concerns seem to have made its way through webboot commits already in the past:
    
    One of them is the following (old, talking about kernel 4.x), touching kernel graphics requirements and kexec (commit log is pretty instructing on how kernel 16->32 bits -> 64 bits works). THIS IS EXACTLY OUR ISSUE HERE:
    u-root/webboot@ac6873c
    
    The other was about setting up kexec to enter kernel through 32 bit entry point to respect kernel asking BIOS for its config:
    u-root/webboot@dfc1429
    tlaurion committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    f1edf70 View commit details
    Browse the repository at this point in the history
  2. modules/linux: have all helper make statements depend on musl-cross-m…

    …ake exposed compiler
    
    linux-qemu.config: have oldconfig format output exposed used compiler
    tlaurion committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    4de01ac View commit details
    Browse the repository at this point in the history
  3. Weird.... deactivating wireless support didnt deactivated 80211.....

    Savagely disabling stuff, not interested in fixing Kconfig dependencies, just want ta working test case here...
    
    Otherwise:
    tail /home/user/heads/build/x86/log/linux.log
    -----
      CC      net/mac80211/chan.o
      CC      net/mac80211/trace.o
      CC      net/mac80211/mlme.o
      CC      net/mac80211/tdls.o
      CC      net/mac80211/ocb.o
      CC      net/mac80211/airtime.o
      CC      net/mac80211/led.o
      CC      net/mac80211/debugfs.o
      CC      net/mac80211/debugfs_sta.o
      CC      net/mac80211/debugfs_netdev.o
      CC      net/mac80211/debugfs_key.o
      CC      net/mac80211/pm.o
      CC      net/mac80211/rc80211_minstrel.o
      CC      net/mac80211/rc80211_minstrel_ht.o
      CC      net/mac80211/rc80211_minstrel_debugfs.o
      CC      net/mac80211/rc80211_minstrel_ht_debugfs.o
      AR      net/mac80211/built-in.a
      AR      net/built-in.a
    make[1]: *** [../Makefile:185: __sub-make] Error 2
    make[1]: Leaving directory '/home/user/heads/build/x86/linux-5.10.5/linux-qemu'
    make: *** [Makefile:419: /home/user/heads/build/x86/linux-5.10.5/linux-qemu/.build] Error 1
    tlaurion committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    de5d83b View commit details
    Browse the repository at this point in the history
  4. linux-qemu.config: Removed CONFIG_EXTRA_FIRMWARE references to iwlwif…

    …i that are unneeded in our test case otherwise
    
    make[6]: *** No rule to make target '../../linux-firmware/iwlwifi-7265D-29.ucode', needed by 'drivers/base/firmware_loader/builtin/iwlwifi-7265D-29.ucode.gen.o'.  Stop.
    tlaurion committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    ea8cfb1 View commit details
    Browse the repository at this point in the history
  5. linux-qemu: add MODULES support and put e1000e compiled as module whi…

    …ch Heads requires
    
    same for usb controller, keyboard and unrelated usb stuff in our use case.
    
    Interesting Kconfig dependency highlighten in that commit, all changes commit from menuconfig...
    tlaurion committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    b1d6e91 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    dcbb11f View commit details
    Browse the repository at this point in the history
  7. Add DRM and FB back, change board KERNEL_ADD to pass vga=791 as per u…

    …-root/webboot@ac6873c. Doesn't change anything. I blame it on qemu at this point
    tlaurion committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    e3efe50 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. Test on real hardware confirms Tinycore is not aimed to kexec properly

    - Tinycore does not support kexec2 (file loading, requires legacy syscall) fallback
    - Booting debian works, need to remove i915 to see if fbsimple can be kicked in and tune basic config with fallbacks if kexec call doesn't work
    tlaurion committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    4d70886 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2023

  1. Going back to more plain config

    - qemu-coreboot-whiptail-tpm1 board config: remove vga-791 statement which is dismissed and should not be specified
    - kexec-boot: remove additional statements to elf.
    - kexec-2.0.26.patch: add more debug stuff on vga setup.
    
    Current state:
    - VESA is not setuped from host kernel detection.
    - kexec call shows : 2012 and other debug codes which are linked to reset-vga codepath (non-working)
    tlaurion committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    d506ef1 View commit details
    Browse the repository at this point in the history
  2. Remove builds from CircleCI at this point. Just loosing compilation t…

    …ime and free resource for nothing on this PR
    tlaurion committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    2207067 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c992996 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2023

  1. kexec2 is not compatible with entry-32bit.... Reverting changes. Tiny…

    …core 14.0 released today, tests with debian shows that again things are ok when drm+gpu replaces fb but not before on both qemu and real hardware
    tlaurion committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    87c7dea View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. Configuration menu
    Copy the full SHA
    bca26c1 View commit details
    Browse the repository at this point in the history