5.4 kernel warning: general protection fault with bpf_probe_read

[diranged]

@etung Edit: all variants with the 5.4 kernel are affected.

Image I'm using:

In our forked repo (with just a change to the CNI networking/instance-type map text file), we built the v1.0.7 tag to try to deal with a possible issue we're seeing with Systemd. The images are not booting though. We booted 3 servers, and all failed with a nearly identical failure:

Git Sha Built: fdb0353

What I expected to happen:

Ideally it would have booted. :)

What actually happened:

All 3 servers booted and failed like this:

[   82.124594] ------------[ cut here ]------------
[   82.127954] General protection fault in user access. Non-canonical address?
[   82.127964] WARNING: CPU: 15 PID: 8953 at arch/x86/mm/extable.c:126 ex_handler_uaccess+0x4d/0x60
[   82.138374] Modules linked in: nf_conntrack_netlink xt_multiport xt_tcpudp veth xt_connmark xt_nat xt_addrtype ipt_rpfilter iptable_raw ip_set_hash_net ip_set nfnetlink cls_bpf algif_hash af_alg sch_ingress iptable_mangle xt_MASQUERADE xt_conntrack xt_comment xt_mark iptable_nat nf_nat iptable_filter ip_tables x_tables overlay squashfs zstd_decompress crc32_pclmul ghash_clmulni_intel aesni_intel loop crypto_simd sch_fq_codel cryptd mousedev psmouse evdev nf_conntrack glue_helper ena button nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel configfs fuse ipv6 crc_ccitt autofs4
[   82.168779] CPU: 15 PID: 8953 Comm: system-probe Not tainted 5.4.95 #1
[   82.172802] Hardware name: Amazon EC2 c5d.4xlarge/, BIOS 1.0 10/16/2017
[   82.176823] RIP: 0010:ex_handler_uaccess+0x4d/0x60
[   82.180200] Code: 83 c4 08 b8 01 00 00 00 5b c3 80 3d 42 05 75 01 00 75 dc 48 c7 c7 60 20 1c 82 48 89 34 24 c6 05 2e 05 75 01 01 e8 51 71 6e 00 <0f> 0b 48 8b 34 24 eb bd 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44
[   82.191514] RSP: 0018:ffffc9000fbe3a58 EFLAGS: 00010282
[   82.195038] RAX: 0000000000000000 RBX: ffffffff81a026d4 RCX: 0000000000000000
[   82.199302] RDX: 000000000000003f RSI: ffffffff82dc365f RDI: 0000000000000246
[   82.203546] RBP: ffffc9000fbe3ab8 R08: 000000131f352426 R09: 000000000000003f
[   82.207785] R10: 000000000000072d R11: 0000000000000000 R12: 0000000000000000
[   82.211996] R13: 000000000000000d R14: 0000000000000000 R15: 0000000000000000
[   82.216187] FS:  00007f9680ff9700(0000) GS:ffff8887dfbc0000(0000) knlGS:0000000000000000
[   82.222115] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   82.225826] CR2: 000000c000849000 CR3: 0000000675c44005 CR4: 00000000007606e0
[   82.230034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   82.234262] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   82.238537] PKRU: 55555554
[   82.241223] Call Trace:
[   82.243690]  fixup_exception+0x45/0x60
[   82.246631]  do_general_protection+0x49/0x150
[   82.249812]  general_protection+0x28/0x30
[   82.252888] RIP: 0010:copy_user_generic_unrolled+0x9e/0xc0
[   82.256511] Code: 48 8d 7f 40 ff c9 75 b6 89 d1 83 e2 07 c1 e9 03 74 12 4c 8b 06 4c 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10 89 d1 <8a> 06 88 07 48 ff c6 48 ff c7 ff c9 75 f2 31 c0 0f 01 ca c3 66 66
[   82.267993] RSP: 0018:ffffc9000fbe3b68 EFLAGS: 00050202
[   82.271533] RAX: 0000000000000001 RBX: ffff888675c90000 RCX: 0000000000000004
[   82.275786] RDX: 0000000000000004 RSI: 0100007f0200007f RDI: ffffc9000fbe3bfc
[   82.280052] RBP: 0000000000000004 R08: 0100007f0200007f R09: 0000000000000008
[   82.284344] R10: 0000000000000004 R11: 00000000d9b007f1 R12: 00007ffffffff000
[   82.288605] R13: 0100007f0200007f R14: ffffc9000fbe3bfc R15: ffff8886f27f03d0
[   82.292897]  __probe_kernel_read+0x54/0x80
[   82.296000]  bpf_probe_read+0x2e/0x60
[   82.298928]  ? tcp_getsockopt+0x5/0x30
[   82.301905]  ? tcp_getsockopt+0x1/0x30
[   82.304866]  ? trace_call_bpf+0x7a/0xf0
[   82.307869]  ? tcp_getsockopt+0x1/0x30
[   82.310880]  ? tcp_getsockopt+0x5/0x30
[   82.313828]  ? kprobe_perf_func+0x252/0x280
[   82.316984]  ? alloc_htab_elem+0x1ac/0x2a0
[   82.320100]  ? tcp_getsockopt+0x1/0x30
[   82.323056]  ? tcp_getsockopt+0x5/0x30
[   82.326005]  ? kprobe_ftrace_handler+0x92/0xf0
[   82.329255]  ? do_tcp_getsockopt.isra.0+0xdc0/0xdc0
[   82.332639]  ? __sys_getsockopt+0x83/0x110
[   82.335746]  ? ftrace_ops_assist_func+0x89/0x110
[   82.339044]  ? __seccomp_filter+0x7b/0x670
[   82.342165]  ? 0xffffffffa04410bf
[   82.344960]  ? do_tcp_getsockopt.isra.0+0xdc0/0xdc0
[   82.348341]  ? tcp_getsockopt+0x1/0x30
[   82.351295]  ? tcp_getsockopt+0x5/0x30
[   82.354377]  ? __sys_getsockopt+0x83/0x110
[   82.357627]  ? __x64_sys_getsockopt+0x21/0x30
[   82.360856]  ? do_syscall_64+0x4e/0xf0
[   82.363853]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   82.367461] ---[ end trace 93b64b599a5e8d08 ]---
[   82.611422] 
[   82.613579] **********************************************************
[   82.617618] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[   82.621694] **                                                      **
[   82.625734] ** trace_printk() being used. Allocating extra memory.  **
[   82.629779] **                                                      **
[   82.633807] ** This means that this is a DEBUG kernel and it is     **
[   82.637839] ** unsafe for production use.                           **
[   82.641867] **                                                      **
[   82.645873] ** If you see this message and you are not debugging    **
[   82.649897] ** the kernel, report this immediately to your vendor!  **
[   82.653968] **                                                      **
[   82.755795] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[   82.759838] **********************************************************

[diranged]

Here's the actual boot log, sorry it wasn't available when I first made the bug report. Also, while the panic happens, the host actually keeps working.. its possible that our other nodes do this, and we just hadn't noticed.

[    0.828884] kvm-stealtime: cpu 2, msr 7df8ac040
[    0.832967]   #3
[    0.673380] kvm-clock: cpu 3, msr 32010c1, secondary cpu clock
[    0.833717] KVM setup async PF for cpu 3
[    0.836884] kvm-stealtime: cpu 3, msr 7df8ec040
[    0.844965]   #4
[    0.673380] kvm-clock: cpu 4, msr 3201101, secondary cpu clock
[    0.845711] KVM setup async PF for cpu 4
[    0.848884] kvm-stealtime: cpu 4, msr 7df92c040
[    0.856961]   #5
[    0.673380] kvm-clock: cpu 5, msr 3201141, secondary cpu clock
[    0.857709] KVM setup async PF for cpu 5
[    0.860884] kvm-stealtime: cpu 5, msr 7df96c040
[    0.864967]   #6
[    0.673380] kvm-clock: cpu 6, msr 3201181, secondary cpu clock
[    0.869427] KVM setup async PF for cpu 6
[    0.872884] kvm-stealtime: cpu 6, msr 7df9ac040
[    0.876963]   #7
[    0.673380] kvm-clock: cpu 7, msr 32011c1, secondary cpu clock
[    0.877715] KVM setup async PF for cpu 7
[    0.880884] kvm-stealtime: cpu 7, msr 7df9ec040
[    0.888963]   #8
[    0.673380] kvm-clock: cpu 8, msr 3201201, secondary cpu clock
[    0.889813] KVM setup async PF for cpu 8
[    0.892884] kvm-stealtime: cpu 8, msr 7dfa2c040
[    0.900948] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    0.904887] TAA CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html for more details.
[    0.908975]   #9
[    0.673380] kvm-clock: cpu 9, msr 3201241, secondary cpu clock
[    0.909718] KVM setup async PF for cpu 9
[    0.912884] kvm-stealtime: cpu 9, msr 7dfa6c040
[    0.916968]  #10
[    0.673380] kvm-clock: cpu 10, msr 3201281, secondary cpu clock
[    0.917697] KVM setup async PF for cpu 10
[    0.920884] kvm-stealtime: cpu 10, msr 7dfaac040
[    0.928966]  #11
[    0.673380] kvm-clock: cpu 11, msr 32012c1, secondary cpu clock
[    0.929710] KVM setup async PF for cpu 11
[    0.932884] kvm-stealtime: cpu 11, msr 7dfaec040
[    0.940971]  #12
[    0.673380] kvm-clock: cpu 12, msr 3201301, secondary cpu clock
[    0.941707] KVM setup async PF for cpu 12
[    0.944884] kvm-stealtime: cpu 12, msr 7dfb2c040
[    0.952964]  #13
[    0.673380] kvm-clock: cpu 13, msr 3201341, secondary cpu clock
[    0.953703] KVM setup async PF for cpu 13
[    0.956884] kvm-stealtime: cpu 13, msr 7dfb6c040
[    0.960962]  #14
[    0.673380] kvm-clock: cpu 14, msr 3201381, secondary cpu clock
[    0.964954] KVM setup async PF for cpu 14
[    0.968884] kvm-stealtime: cpu 14, msr 7dfbac040
[    0.972966]  #15
[    0.673380] kvm-clock: cpu 15, msr 32013c1, secondary cpu clock
[    0.973743] KVM setup async PF for cpu 15
[    0.976884] kvm-stealtime: cpu 15, msr 7dfbec040
[    0.984915] smp: Brought up 1 node, 16 CPUs
[    0.988887] smpboot: Max logical packages: 1
[    0.992051] smpboot: Total of 16 processors activated (96000.00 BogoMIPS)
[    1.061396] node 0 initialised, 7162892 pages in 68ms
[    1.065584] devtmpfs: initialized
[    1.068917] x86/mm: Memory block size: 128MB
[    1.074557] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.080892] futex hash table entries: 4096 (order: 6, 262144 bytes, linear)
[    1.085105] NET: Registered protocol family 16
[    1.088430] audit: initializing netlink subsys (disabled)
[    1.088899] audit: type=2000 audit(1617231809.809:1): state=initialized audit_enabled=0 res=1
[    1.096890] cpuidle: using governor ladder
[    1.100897] cpuidle: using governor menu
[    1.105012] ACPI: bus type PCI registered
[    1.108100] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    1.109020] PCI: Using configuration type 1 for base access
[    1.117128] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.120889] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.232919] ACPI: Added _OSI(Module Device)
[    1.232919] ACPI: Added _OSI(Processor Device)
[    1.236138] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.240887] ACPI: Added _OSI(Processor Aggregator Device)
[    1.244515] ACPI: Added _OSI(Linux-Dell-Video)
[    1.248887] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    1.252486] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    1.257595] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    1.262037] ACPI: Interpreter enabled
[    1.264894] ACPI: (supports S0 S4 S5)
[    1.267858] ACPI: Using IOAPIC for interrupt routing
[    1.268898] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.277209] ACPI: Enabled 16 GPEs in block 00 to 0F
[    1.284900] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    1.288890] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3]
[    1.292893] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    1.301128] acpiphp: Slot [3] registered
[    1.304188] acpiphp: Slot [4] registered
[    1.308905] acpiphp: Slot [5] registered
[    1.312015] acpiphp: Slot [6] registered
[    1.312903] acpiphp: Slot [7] registered
[    1.315954] acpiphp: Slot [8] registered
[    1.320902] acpiphp: Slot [9] registered
[    1.323985] acpiphp: Slot [10] registered
[    1.324903] acpiphp: Slot [11] registered
[    1.328886] acpiphp: Slot [12] registered
[    1.332903] acpiphp: Slot [13] registered
[    1.336005] acpiphp: Slot [14] registered
[    1.336902] acpiphp: Slot [15] registered
[    1.340886] acpiphp: Slot [16] registered
[    1.344903] acpiphp: Slot [17] registered
[    1.348018] acpiphp: Slot [18] registered
[    1.348902] acpiphp: Slot [19] registered
[    1.352887] acpiphp: Slot [20] registered
[    1.356904] acpiphp: Slot [21] registered
[    1.360020] acpiphp: Slot [22] registered
[    1.360903] acpiphp: Slot [23] registered
[    1.364886] acpiphp: Slot [24] registered
[    1.368902] acpiphp: Slot [25] registered
[    1.371983] acpiphp: Slot [26] registered
[    1.376902] acpiphp: Slot [27] registered
[    1.379985] acpiphp: Slot [28] registered
[    1.380903] acpiphp: Slot [29] registered
[    1.384000] acpiphp: Slot [30] registered
[    1.388903] acpiphp: Slot [31] registered
[    1.391985] PCI host bridge to bus 0000:00
[    1.392888] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    1.396887] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    1.400887] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    1.408887] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[    1.412887] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.416562] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    1.421337] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    1.426142] pci 0000:00:01.3: [8086:7113] type 00 class 0x000000
[    1.429890] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    1.436907] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
[    1.440955] pci 0000:00:01.3: PIIX4 devres E PIO at fff0-ffff
[    1.444814] pci 0000:00:01.3: PIIX4 devres F MMIO at ffc00000-ffffffff
[    1.448907] pci 0000:00:01.3: PIIX4 devres G PIO at fff0-ffff
[    1.452702] pci 0000:00:01.3: PIIX4 devres H MMIO at ffc00000-ffffffff
[    1.456903] pci 0000:00:01.3: PIIX4 devres I PIO at fff0-ffff
[    1.460645] pci 0000:00:01.3: PIIX4 devres J PIO at fff0-ffff
[    1.464890] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x170 took 31250 usecs
[    1.469182] pci 0000:00:03.0: [1d0f:1111] type 00 class 0x030000
[    1.473294] pci 0000:00:03.0: reg 0x10: [mem 0xfe400000-0xfe7fffff pref]
[    1.477998] pci 0000:00:03.0: reg 0x30: [mem 0xfebd0000-0xfebdffff pref]
[    1.481192] pci 0000:00:04.0: [1d0f:8061] type 00 class 0x010802
[    1.486204] pci 0000:00:04.0: reg 0x10: [mem 0xfebf0000-0xfebf3fff]
[    1.498340] pci 0000:00:05.0: [1d0f:ec20] type 00 class 0x020000
[    1.501778] pci 0000:00:05.0: reg 0x10: [mem 0xfebf4000-0xfebf7fff]
[    1.506440] pci 0000:00:05.0: reg 0x18: [mem 0xfe800000-0xfe8fffff pref]
[    1.514440] pci 0000:00:05.0: reg 0x20: [mem 0xfebe0000-0xfebeffff]
[    1.519147] pci 0000:00:1e.0: [1d0f:8061] type 00 class 0x010802
[    1.525725] pci 0000:00:1e.0: reg 0x10: [mem 0xfebf8000-0xfebfbfff]
[    1.533897] pci 0000:00:1f.0: [1d0f:cd01] type 00 class 0x010802
[    1.537375] pci 0000:00:1f.0: reg 0x10: [mem 0xfebfc000-0xfebfffff]
[    1.541633] pci 0000:00:1f.0: reg 0x18: [mem 0xfe900000-0xfe901fff pref]
[    1.548885] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[    1.552980] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    1.556779] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    1.560982] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[    1.564726] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
[    1.569521] iommu: Default domain type: Translated 
[    1.572912] pci 0000:00:03.0: vgaarb: setting as boot VGA device
[    1.576837] pci 0000:00:03.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.580894] pci 0000:00:03.0: vgaarb: bridge control possible
[    1.584642] vgaarb: loaded
[    1.684986] EDAC MC: Ver: 3.0.0
[    1.688899] PCI: Using ACPI for IRQ routing
[    1.693162] NetLabel: Initializing
[    1.696092] NetLabel:  domain hash size = 128
[    1.696887] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    1.700645] NetLabel:  unlabeled traffic allowed by default
[    1.704981] clocksource: Switched to clocksource kvm-clock
[    1.716722] *** VALIDATE bpf ***
[    1.719572] VFS: Disk quotas dquot_6.6.0
[    1.722650] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.726827] *** VALIDATE ramfs ***
[    1.729694] *** VALIDATE hugetlbfs ***
[    1.732714] pnp: PnP ACPI init
[    1.735753] pnp: PnP ACPI: found 5 devices
[    1.740447] thermal_sys: Registered thermal governor 'fair_share'
[    1.740448] thermal_sys: Registered thermal governor 'step_wise'
[    1.744346] thermal_sys: Registered thermal governor 'user_space'
[    1.752836] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    1.763038] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    1.766959] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    1.770919] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    1.775105] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
[    1.779320] NET: Registered protocol family 2
[    1.782733] tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    1.788998] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    1.795216] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    1.801105] TCP: Hash tables configured (established 262144 bind 65536)
[    1.805261] UDP hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    1.809473] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    1.815295] NET: Registered protocol family 1
[    1.818500] NET: Registered protocol family 44
[    1.821761] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.825580] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    1.829567] pci 0000:00:03.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.835786] PCI: CLS 0 bytes, default 64
[    1.838859] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.842865] software IO TLB: mapped [mem 0xbbff9000-0xbfff9000] (64MB)
[    1.846970] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2b3e459bf4c, max_idle_ns: 440795289890 ns
[    1.853756] clocksource: Switched to clocksource tsc
[    1.858510] check: Scanning for low memory corruption every 60 seconds
[    1.865130] Initialise system trusted keyrings
[    1.868401] Key type blacklist registered
[    1.871526] workingset: timestamp_bits=36 max_order=23 bucket_order=0
[    1.876810] zbud: loaded
[    1.882202] Key type asymmetric registered
[    1.885304] Asymmetric key parser 'x509' registered
[    1.888741] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.894533] io scheduler mq-deadline registered
[    1.897839] io scheduler kyber registered
[    1.900963] io scheduler bfq registered
[    1.904211] crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
[    1.907681] crc32: self tests passed, processed 225944 bytes in 124748 nsec
[    1.912066] crc32c: CRC_LE_BITS = 64
[    1.914970] crc32c: self tests passed, processed 225944 bytes in 63112 nsec
[    1.928375] crc32_combine: 8373 self tests passed
[    1.940980] crc32c_combine: 8373 self tests passed
[    1.944566] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.949158] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.980080] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.987105] nvme nvme0: pci function 0000:00:04.0
[    1.990480] nvme nvme1: pci function 0000:00:1e.0
[    1.990674] PCI Interrupt Link [LNKD] enabled at IRQ 11
[    1.993857] nvme nvme2: pci function 0000:00:1f.0
[    1.997510] PCI Interrupt Link [LNKB] enabled at IRQ 10
[    2.001060] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    2.010757] i8042: Warning: Keylock active
[    2.014972] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.018394] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.021953] rtc_cmos 00:00: RTC can wake from S4
[    2.025993] rtc_cmos 00:00: registered as rtc0
[    2.029258] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
[    2.033215] device-mapper: uevent: version 1.0.3
[    2.036561] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: [email protected]
[    2.042775] intel_pstate: P-states controlled by the platform
[    2.046542] hidraw: raw HID events driver (C) Jiri Kosina
[    2.050232] NET: Registered protocol family 17
[    2.054661] IPI shorthand broadcast: enabled
[    2.057858] sched_clock: Marking stable (1388465926, 669380668)->(2212002952, -154156358)
[    2.064014] registered taskstats version 1
[    2.067121] Loading compiled-in X.509 certificates
[    2.071164] Loaded X.509 cert 'Build time autogenerated kernel key: 09e1f18a9990a2f7945672f523c23ba69107e47a'
[    2.077935] zswap: loaded using pool lzo/zbud
[    2.081175] ima: No TPM chip found, activating TPM-bypass!
[    2.084800] ima: Allocated hash algorithm: sha1
[    2.088102] ima: No architecture policies found
[    2.091698] rtc_cmos 00:00: setting system clock to 2021-03-31T23:03:31 UTC (1617231811)
[    2.097663] device-mapper: init: waiting for all devices to be available before creating mapped devices
[    2.112446] nvme nvme2: 16/0/0 default/read/poll queues
[    2.114376] nvme nvme1: 2/0/0 default/read/poll queues
[    2.214558] nvme nvme0: 2/0/0 default/read/poll queues
[    2.220568]  nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10
[    2.284282] GPT:Primary header thinks Alt. header is not at the end of the disk.
[    2.290063] GPT:2097151 != 104857599
[    2.293015] GPT:Alternate GPT header not at the end of the disk.
[    2.296843] GPT:2097151 != 104857599
[    2.299765] GPT: Use GNU Parted to correct GPT errors.
[    2.303296]  nvme1n1: p1
[    2.553384] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    2.559848] device-mapper: verity: sha256 using implementation "sha256-avx2"
[    2.564243] device-mapper: table: 252:0: adding target device (start sect 0 len 1884160) caused an alignment inconsistency
[    2.571454] device-mapper: table: 252:0: adding target device (start sect 0 len 1884160) caused an alignment inconsistency
[    2.578580] device-mapper: ioctl: dm-0 (root) is ready
[    2.582816] md: Waiting for all devices to be available before autodetect
[    2.586949] md: If you don't use raid, use raid=noautodetect
[    2.590738] md: Autodetecting RAID arrays.
[    2.593843] md: autorun ...
[    2.596446] md: ... autorun DONE.
[    2.753701] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null)
[    2.757992] VFS: Mounted root (ext4 filesystem) readonly on device 252:0.
[    2.811136] devtmpfs: mounted
[    2.814695] Freeing unused decrypted memory: 2040K
[    2.818673] Freeing unused kernel image memory: 2308K
[    2.822210] Write protecting the kernel read-only data: 22528k
[    2.826568] Freeing unused kernel image memory: 2020K
[    2.830182] Freeing unused kernel image memory: 184K
[    2.833654] Run /sbin/init as init process
[    5.651166] SELinux:  policy capability network_peer_controls=1
[    5.654956] SELinux:  policy capability open_perms=1
[    5.658355] SELinux:  policy capability extended_socket_class=1
[    5.662146] SELinux:  policy capability always_check_network=0
[    5.665871] SELinux:  policy capability cgroup_seclabel=1
[    5.669448] SELinux:  policy capability nnp_nosuid_transition=1
[    5.689984] audit: type=1403 audit(1617231815.096:2): auid=4294967295 ses=4294967295 lsm=selinux res=1
[    5.690247] systemd[1]: Successfully loaded SELinux policy in 221.829ms.
[    6.324788] systemd[1]: Inserted module 'autofs4'
[    6.714603] NET: Registered protocol family 10
[    6.830544] Segment Routing with IPv6
[    6.837027] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 911us.
[    6.845042] systemd[1]: systemd 247 running in system mode. (-PAM -AUDIT +SELINUX -IMA -APPARMOR -SMACK +SYSVINIT -UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL -XZ -LZ4 -ZSTD +SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid)
[    6.876928] systemd[1]: Detected virtualization kvm.
[    6.880393] systemd[1]: Detected architecture x86-64.
[    6.883860] systemd[1]: Detected first boot.

Welcome to ND-Bottlerocket OS $CODEBUILD_SOURCE_VERSION 1.0.7!

[    6.903061] systemd[1]: No hostname configured.
[    6.906333] systemd[1]: Set hostname to <localhost>.
[    6.910921] systemd[1]: Initializing machine ID from KVM UUID.
[    7.104605] systemd[1]: Populated /etc with preset unit settings.
[    7.137109] systemd[1]: Queued start job for default target Graphical Interface.
[    7.151809] systemd[1]: Created slice system-modprobe.slice.
[  OK  ] Created slice system-modprobe.slice.
[    7.156250] systemd[1]: Reached target Login Prompts.
[  OK  ] Reached target Login Prompts.
[    7.160287] systemd[1]: Reached target Paths.
[  OK  ] Reached target Paths.
[    7.163950] systemd[1]: Reached target Remote File Systems.
[  OK  ] Reached target Remote File Systems.
[    7.168211] systemd[1]: Reached target Slices.
[  OK  ] Reached target Slices.
[    7.171962] systemd[1]: Reached target Swap.
[  OK  ] Reached target Swap.
[    7.175616] systemd[1]: Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[    7.180534] systemd[1]: Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Audit Socket.
[    7.184879] systemd[1]: Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket (/dev/log).
[    7.189462] systemd[1]: Listening on Journal Socket.
[  OK  ] Listening on Journal Socket.
[    7.193577] systemd[1]: Listening on udev Control Socket.
[  OK  ] Listening on udev Control Socket.
[    7.197863] systemd[1]: Listening on udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[    7.202716] systemd[1]: Mounting Huge Pages File System...
         Mounting Huge Pages File System...
[    7.207624] systemd[1]: Mounting POSIX Message Queue File System...
         Mounting POSIX Message Queue File System...
[    7.213022] systemd[1]: Mounting CNI Configuration Directory (/etc/cni)...
         Mounting CNI Configuration Directory (/etc/cni)...
[    7.218716] systemd[1]: Mounting Kernel Debug File System...
         Mounting Kernel Debug File System...
[    7.223742] systemd[1]: Mounting Kernel Trace File System...
         Mounting Kernel Trace File System...
[    7.229135] systemd[1]: Mounting Temporary Directory (/tmp)...
         Mounting Temporary Directory (/tmp)...
[    7.331021] systemd[1]: Starting Load audit rules...
         Starting Load audit rules...
[    7.335845] systemd[1]: Starting Create list of static device nodes for the current kernel...
         Starting Create list of static device nodes for the current kernel...
[    7.343676] systemd[1]: Starting Load Kernel Module configfs...
         Starting Load Kernel Module configfs...
[    7.348730] systemd[1]: Starting Load Kernel Module fuse...
         Starting Load Kernel Module fuse...
[    7.353722] systemd[1]: Starting Copy SELinux policy files...
         Starting Copy SELinux policy files...
[    7.358236] fuse: init (API version 7.31)
[    7.361365] *** VALIDATE fuse ***
[    7.364129] *** VALIDATE fuse ***
[    7.367285] systemd[1]: Starting Journal Service...
         Starting Journal Service...
[    7.372435] systemd[1]: Starting Load Kernel Modules...
         Starting Load Kernel Modules...
[    7.377327] systemd[1]: Starting Remount Root and Kernel File Systems...
         Starting Remount Root and Kernel File Systems...
[    7.382336] systemd[1]: Condition check resulted in Repartition Root Disk being skipped.
[    7.388999] systemd[1]: Starting Coldplug All udev Devices...
         Starting Coldplug All udev Devices...
[    7.394691] systemd[1]: Mounted Huge Pages File System.
[  OK  ] Mounted Huge Pages File System.
[    7.398919] systemd[1]: Mounted POSIX Message Queue File System.
[  OK  ] Mounted POSIX Message Queue File System.
[    7.403575] systemd[1]: Mounted CNI Configuration Directory (/etc/cni).
[  OK  ] Mounted CNI Configuration Directory (/etc/cni).
[    7.408543] systemd[1]: Started Journal Service.
[  OK  ] Started Journal Service.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted Kernel Trace File System.
[  OK  ] Mounted Temporary Directory (/tmp).
[  OK  ] Finished Load audit rules.
[  OK  ] Finished Create list of static device nodes for the current kernel.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Mounting FUSE Control File System...
         Starting Create System Users...
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Finished Load Kernel Module configfs.
         Mounting Kernel Configuration File System...
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
         Mounting License files...
         Starting Rule-based Manager for Device Events and Files...
[  OK  ] Finished Copy SELinux policy files.
[  OK  ] Started Rule-based Manager for Device Events and Files.
[    7.647961] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[  OK  ] Finishe[    7.654110] ACPI: Power Button [PWRF]
d Coldplug All udev Devices.
[    7.657588] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input4
[    7.663733] ACPI: Sleep Button [SLPF]
         Starting Wait for udev To Complete Device Initialization...
[    7.667466] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.4.0g
[    7.681676] ena 0000:00:05.0: ENA device version: 0.10
[    7.685162] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[  OK  ] Finished Load Kernel Modules.
         Starting Apply Kernel Variables...
[    7.774577] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 02:2c:d8:1f:ee:f5
[    7.806856] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input5
[    7.813172] mousedev: PS/2 mouse device common for all mice
[    7.877466] cryptd: max_cpu_qlen set to 1000
[  OK  ] Finished Apply Kernel Variables.
[    7.989343] AVX2 version of gcm_enc/dec engaged.
[    7.992631] AES CTR mode by8 optimization enabled
[  OK  ] Found device Amazon Elastic Block Store BOTTLEROCKET-PRIVATE.
[  OK  ] Found device Amazon Elastic Block Store BOTTLEROCKET-DATA.
         Starting Prepare Local Directory (/local)...
[    8.197183] loop: module loaded
[    8.318970]  nvme1n1: p1
[    8.423775]  nvme1n1: p1
[    8.481618] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[  OK  ] Mounted License files.
[    8.530474] EXT4-fs (nvme1n1p1): mounted filesystem with ordered data mode. Opts: 
[    8.540634] EXT4-fs (nvme1n1p1): resizing filesystem from 261632 to 13106688 blocks
[    8.822488] EXT4-fs (nvme1n1p1): resized filesystem to 13106688
[  OK  ] Finished Prepare Local Directory (/local).
         Mounting Opt Directory (/opt)...
         Mounting Var Directory (/var)...
[  OK  ] Mounted Opt Directory (/opt).
[  OK  ] Mounted Var Directory (/var).
         Mounting Private Directory (/var/lib/bottlerocket)...
         Mounting Kernel Development Sources (Read-Only)...
         Starting Flush Journal to Persistent Storage...
[    8.945050] EXT4-fs (nvme0n1p10): mounted filesystem with ordered data mode. Opts: 
         Starting Load/Save Random Seed...
[    8.953285] systemd-journald[1247]: Received client request to flush runtime journal.
[  OK  ] Mounted Private Directory (/var/lib/bottlerocket).
[  OK  ] Finished Load/Save Random Seed.
[  OK  ] Reached target First Boot Complete.
[  OK  ] Mounted Kernel Development Sources (Read-Only).
         Mounting Kernel Development Sources (Read-Write)...
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Mounted Kernel Development Sources (Read-Write).
[  OK  ] Reached target Local File Systems.
         Starting Rebuild Dynamic Linker Cache...
         Starting Commit a transient machine-id on disk...
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Commit a transient machine-id on disk.
[  OK  ] Finished Wait for udev To Complete Device Initialization.
[  OK  ] Finished Rebuild Dynamic Linker Cache.
[*     ] A start job is running for Create V�es and Directories (4s / no limit)
�M
�[K[**    ] A start job is running for Create V�es and Directories (5s / no limit)
�M
�[K[***   ] A start job is running for Create V�es and Directories (6s / no limit)
�M
�[K[ ***  ] A start job is running for Create V�es and Directories (6s / no limit)
�M
�[K[  *** ] A start job is running for Create V�es and Directories (7s / no limit)
�M
�[K[   ***] A start job is running for Create V�es and Directories (7s / no limit)
�M
�[K[  OK  ] Finished Create Volatile Files and Directories.
�[K         Starting Rebuild Journal Catalog...
[  OK  ] Finished Rebuild Journal Catalog.
         Starting Update is Completed...
[  OK  ] Finished Update is Completed.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Scheduled Metricdog Pings.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Reached target Basic System.
         Starting ACPI event daemon...
         Starting Bottlerocket data store migrator...
         Starting wicked AutoIPv4 supplicant service...
         Starting wicked DHCPv4 supplicant service...
         Starting wicked DHCPv6 supplicant service...
[  OK  ] Started ACPI event daemon.
[  OK  ] Started wicked AutoIPv4 supplicant service.
[  OK  ] Started wicked DHCPv4 supplicant service.
[  OK  ] Started wicked DHCPv6 supplicant service.
         Starting wicked network management service daemon...
[  OK  ] Started wicked network management service daemon.
         Starting wicked network nanny service...
[  OK  ] Started wicked network nanny service.
         Starting wicked managed network interfaces...
[   15.012086] ena 0000:00:05.0 eth0: Local page cache is disabled for less than 16 channels
[   14.472071] migrator[3073]: Data store does not exist at given path, exiting (/var/lib/bottlerocket/datastore/current)
[  OK  ] Finished Bottlerocket data store migrator.
         Starting Datastore creator...
[  OK  ] Finished Datastore creator.
         Starting Bottlerocket API server...
[  OK  ] Started Bottlerocket API server.
[  OK  ] Finished wicked managed network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting Bottlerocket userdata configuration system...
[  OK  ] Finished Bottlerocket userdata configuration system.
         Starting User-specified setting generators...
[  OK  ] Finished User-specified setting generators.
         Starting Applies settings to create config files...
[  OK  ] Created slice system-host\x2dcontainers.slice.
[   19.170055] ip_local_port_range: prefer different parity for start/end values.
[  OK  ] Finished Applies settings to create config files.
[  OK  ] Reached target Bottlerocket user and dynamic configuration complete.
[  OK  ] Started A versatile implementation of the Network Time Protocol.
         Starting containerd container runtime...
         Starting containerd runtime for host containers...
[  OK  ] Started containerd runtime for host containers.
         Starting Host container: admin...
         Starting Host container: control...
[  OK  ] Started containerd container runtime.
[  OK  ] Started Host container: admin.
[  OK  ] Started Host container: control.
         Starting Kubelet...
[  OK  ] Started Kubernetes systemd probe.
[  OK  ] Started Kubelet.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Call signpost to mark the �r all required targets are met....
[   32.394071] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[   31.897640] host-ctr[4006]: Failed to parse trusted_user_ca_keys from /.bottlerocket/host-containers/admin/user-data
[   31.985640] host-ctr[4006]: Server listening on 0.0.0.0 port 22.
[   31.986742] host-ctr[4006]: Server listening on :: port 22.
[  OK  ] Finished Call signpost to mark the �ter all required targets are met..
[   46.183269] NET: Registered protocol family 38
[   53.829569] pci 0000:00:06.0: [1d0f:ec20] type 00 class 0x020000
[   53.833896] pci 0000:00:06.0: reg 0x10: [mem 0x00000000-0x00003fff]
[   53.838386] pci 0000:00:06.0: reg 0x18: [mem 0x00000000-0x000fffff pref]
[   53.843022] pci 0000:00:06.0: reg 0x20: [mem 0x00000000-0x0000ffff]
[   53.848254] pci 0000:00:06.0: BAR 2: assigned [mem 0xc0000000-0xc00fffff pref]
[   53.854451] pci 0000:00:06.0: BAR 4: assigned [mem 0xc0100000-0xc010ffff]
[   53.858933] pci 0000:00:06.0: BAR 0: assigned [mem 0xc0110000-0xc0113fff]
[   53.863455] ena 0000:00:06.0: enabling device (0000 -> 0002)
[   53.878896] ena 0000:00:06.0: ENA device version: 0.10
[   53.882707] ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
[   53.990924] ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem c0110000, mac addr 02:43:c4:d5:4f:d5
[   60.562840] ena 0000:00:06.0 eth1: Local page cache is disabled for less than 16 channels
[   63.281557] IPv6: ADDRCONF(NETDEV_CHANGE): enidc76012d366: link becomes ready
[   63.286033] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   63.426396] enidc76012d366: Caught tx_queue_len zero misconfig
[   64.466450] IPv6: ADDRCONF(NETDEV_CHANGE): enie061eef35a4: link becomes ready
[   64.470803] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   64.619719] enie061eef35a4: Caught tx_queue_len zero misconfig
[   65.608134] IPv6: ADDRCONF(NETDEV_CHANGE): enif9d52485c99: link becomes ready
[   65.612531] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   65.768707] enif9d52485c99: Caught tx_queue_len zero misconfig
[   67.719184] pci 0000:00:07.0: [1d0f:ec20] type 00 class 0x020000
[   67.723188] pci 0000:00:07.0: reg 0x10: [mem 0x00000000-0x00003fff]
[   67.727243] pci 0000:00:07.0: reg 0x18: [mem 0x00000000-0x000fffff pref]
[   67.731471] pci 0000:00:07.0: reg 0x20: [mem 0x00000000-0x0000ffff]
[   67.736297] pci 0000:00:07.0: BAR 2: assigned [mem 0xc0200000-0xc02fffff pref]
[   67.742333] pci 0000:00:07.0: BAR 4: assigned [mem 0xc0120000-0xc012ffff]
[   67.746510] pci 0000:00:07.0: BAR 0: assigned [mem 0xc0114000-0xc0117fff]
[   67.750958] ena 0000:00:07.0: enabling device (0000 -> 0002)
[   67.766277] ena 0000:00:07.0: ENA device version: 0.10
[   67.769812] ena 0000:00:07.0: ENA controller version: 0.0.1 implementation version 1
[   67.872705] ena 0000:00:07.0: Elastic Network Adapter (ENA) found at mem c0114000, mac addr 02:3d:e3:7e:f6:63
[   75.010785] ena 0000:00:07.0 eth2: Local page cache is disabled for less than 16 channels
[   82.124594] ------------[ cut here ]------------
[   82.127954] General protection fault in user access. Non-canonical address?
[   82.127964] WARNING: CPU: 15 PID: 8953 at arch/x86/mm/extable.c:126 ex_handler_uaccess+0x4d/0x60
[   82.138374] Modules linked in: nf_conntrack_netlink xt_multiport xt_tcpudp veth xt_connmark xt_nat xt_addrtype ipt_rpfilter iptable_raw ip_set_hash_net ip_set nfnetlink cls_bpf algif_hash af_alg sch_ingress iptable_mangle xt_MASQUERADE xt_conntrack xt_comment xt_mark iptable_nat nf_nat iptable_filter ip_tables x_tables overlay squashfs zstd_decompress crc32_pclmul ghash_clmulni_intel aesni_intel loop crypto_simd sch_fq_codel cryptd mousedev psmouse evdev nf_conntrack glue_helper ena button nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel configfs fuse ipv6 crc_ccitt autofs4
[   82.168779] CPU: 15 PID: 8953 Comm: system-probe Not tainted 5.4.95 #1
[   82.172802] Hardware name: Amazon EC2 c5d.4xlarge/, BIOS 1.0 10/16/2017
[   82.176823] RIP: 0010:ex_handler_uaccess+0x4d/0x60
[   82.180200] Code: 83 c4 08 b8 01 00 00 00 5b c3 80 3d 42 05 75 01 00 75 dc 48 c7 c7 60 20 1c 82 48 89 34 24 c6 05 2e 05 75 01 01 e8 51 71 6e 00 <0f> 0b 48 8b 34 24 eb bd 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44
[   82.191514] RSP: 0018:ffffc9000fbe3a58 EFLAGS: 00010282
[   82.195038] RAX: 0000000000000000 RBX: ffffffff81a026d4 RCX: 0000000000000000
[   82.199302] RDX: 000000000000003f RSI: ffffffff82dc365f RDI: 0000000000000246
[   82.203546] RBP: ffffc9000fbe3ab8 R08: 000000131f352426 R09: 000000000000003f
[   82.207785] R10: 000000000000072d R11: 0000000000000000 R12: 0000000000000000
[   82.211996] R13: 000000000000000d R14: 0000000000000000 R15: 0000000000000000
[   82.216187] FS:  00007f9680ff9700(0000) GS:ffff8887dfbc0000(0000) knlGS:0000000000000000
[   82.222115] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   82.225826] CR2: 000000c000849000 CR3: 0000000675c44005 CR4: 00000000007606e0
[   82.230034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   82.234262] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   82.238537] PKRU: 55555554
[   82.241223] Call Trace:
[   82.243690]  fixup_exception+0x45/0x60
[   82.246631]  do_general_protection+0x49/0x150
[   82.249812]  general_protection+0x28/0x30
[   82.252888] RIP: 0010:copy_user_generic_unrolled+0x9e/0xc0
[   82.256511] Code: 48 8d 7f 40 ff c9 75 b6 89 d1 83 e2 07 c1 e9 03 74 12 4c 8b 06 4c 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10 89 d1 <8a> 06 88 07 48 ff c6 48 ff c7 ff c9 75 f2 31 c0 0f 01 ca c3 66 66
[   82.267993] RSP: 0018:ffffc9000fbe3b68 EFLAGS: 00050202
[   82.271533] RAX: 0000000000000001 RBX: ffff888675c90000 RCX: 0000000000000004
[   82.275786] RDX: 0000000000000004 RSI: 0100007f0200007f RDI: ffffc9000fbe3bfc
[   82.280052] RBP: 0000000000000004 R08: 0100007f0200007f R09: 0000000000000008
[   82.284344] R10: 0000000000000004 R11: 00000000d9b007f1 R12: 00007ffffffff000
[   82.288605] R13: 0100007f0200007f R14: ffffc9000fbe3bfc R15: ffff8886f27f03d0
[   82.292897]  __probe_kernel_read+0x54/0x80
[   82.296000]  bpf_probe_read+0x2e/0x60
[   82.298928]  ? tcp_getsockopt+0x5/0x30
[   82.301905]  ? tcp_getsockopt+0x1/0x30
[   82.304866]  ? trace_call_bpf+0x7a/0xf0
[   82.307869]  ? tcp_getsockopt+0x1/0x30
[   82.310880]  ? tcp_getsockopt+0x5/0x30
[   82.313828]  ? kprobe_perf_func+0x252/0x280
[   82.316984]  ? alloc_htab_elem+0x1ac/0x2a0
[   82.320100]  ? tcp_getsockopt+0x1/0x30
[   82.323056]  ? tcp_getsockopt+0x5/0x30
[   82.326005]  ? kprobe_ftrace_handler+0x92/0xf0
[   82.329255]  ? do_tcp_getsockopt.isra.0+0xdc0/0xdc0
[   82.332639]  ? __sys_getsockopt+0x83/0x110
[   82.335746]  ? ftrace_ops_assist_func+0x89/0x110
[   82.339044]  ? __seccomp_filter+0x7b/0x670
[   82.342165]  ? 0xffffffffa04410bf
[   82.344960]  ? do_tcp_getsockopt.isra.0+0xdc0/0xdc0
[   82.348341]  ? tcp_getsockopt+0x1/0x30
[   82.351295]  ? tcp_getsockopt+0x5/0x30
[   82.354377]  ? __sys_getsockopt+0x83/0x110
[   82.357627]  ? __x64_sys_getsockopt+0x21/0x30
[   82.360856]  ? do_syscall_64+0x4e/0xf0
[   82.363853]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   82.367461] ---[ end trace 93b64b599a5e8d08 ]---
[   82.611422] 
[   82.613579] **********************************************************
[   82.617618] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[   82.621694] **                                                      **
[   82.625734] ** trace_printk() being used. Allocating extra memory.  **
[   82.629779] **                                                      **
[   82.633807] ** This means that this is a DEBUG kernel and it is     **
[   82.637839] ** unsafe for production use.                           **
[   82.641867] **                                                      **
[   82.645873] ** If you see this message and you are not debugging    **
[   82.649897] ** the kernel, report this immediately to your vendor!  **
[   82.653968] **                                                      **
[   82.755795] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[   82.759838] **********************************************************

[bcressey]

I've shared this report with our kernel team.

Are you using one of the BPF CNI providers like Calico or Cilium?

Regarding the trace_printk() warning - did you make any changes to the kernel or its configuration? You didn't mention that, so I'm guessing not, but want to confirm.

[diranged]

I've shared this report with our kernel team.

Are you using one of the BPF CNI providers like Calico or Cilium?

We are using the Calico CNI in eBPF mode along with the AWS VPC CNI networking setup..

Regarding the trace_printk() warning - did you make any changes to the kernel or its configuration? You didn't mention that, so I'm guessing not, but want to confirm.

We did not add anything that I knw of that would cause this..

[bcressey]

Thanks, that should help us repro & verify a fix once we understand the issue.

[diranged]

For what its worth - turning off eBPF does not disable this warning message. I am not sure why.

[etungsten]

Hi @diranged, I have a quick update to share. We're still working on this internally with our kernel team. In the meantime, we're releasing a Kubernetes 1.20 variant with our v1.1.0 release that uses the 5.10 kernel, where we believe this issue is fixed. If you're able to upgrade, we'd recommend that as the best option.

If you wish to continue using your current variant, you can also create a custom build that swaps out the 5.4 kernel for the 5.10 kernel in the variant of your choice. Here’s an example of how to achieve that: 906df655.

[diranged]

Hi @diranged, I have a quick update to share. We're still working on this internally with our kernel team. In the meantime, we're releasing a Kubernetes 1.20 variant with our v1.1.0 release that uses the 5.10 kernel, where we believe this issue is fixed. If you're able to upgrade, we'd recommend that as the best option.

If you wish to continue using your current variant, you can also create a custom build that swaps out the 5.4 kernel for the 5.10 kernel in the variant of your choice. Here’s an example of how to achieve that: 906df65.

I saw the 1.1.0 release.. looks like we need to wait for EKS 1.20 (coming soon it seems) right? We have to upgrade the cluster before we can upgrade our nodes, I believe.

[tjkirch]

I saw the 1.1.0 release.. looks like we need to wait for EKS 1.20 (coming soon it seems) right? We have to upgrade the cluster before we can upgrade our nodes, I believe.

EKS has now released 1.20 support! 🎉

[diranged]

👍 we're in the upgrade process now.. and i've tested the 1.1.1 release w/ eks 1.20 and no longer see the kernel issue on-startup! yay!

[etungsten]

Hi @diranged, I'm glad to hear that you're no longer running into the issue.

Although this is fixed in variants with the 5.10 kernel, other variants with the 5.4 kernel will still have the issue. So I would like to edit this issue to track the fix for our other 5.4 kernel variants.

Feel free to unsubscribe from the issue notification if it gets too noisy.

[stmcginnis]

We have since moved past this kernel version and don't believe it is still an issue. Closing, but feel free to reopen a new issue if there is anything else related.