Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BPF] failed to dump the conntrack table #9651

Open
ioworker0 opened this issue Dec 30, 2024 · 10 comments · May be fixed by #9704
Open

[BPF] failed to dump the conntrack table #9651

ioworker0 opened this issue Dec 30, 2024 · 10 comments · May be fixed by #9704
Labels
area/bpf eBPF Dataplane issues kind/bug

Comments

@ioworker0
Copy link
Contributor

version: v3.27.3
dataplane: eBPF

I modified the bpfMapSizeConntrack parameter in the global Felix configuration and then deleted a calico-node. After that, the following errors occurred:

...
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:25, Iface:"cali3d68ded7758", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:27, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:17, Iface:"cali681df24beb0", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:19, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:29, Iface:"calicf78b17719a", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:31, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:22, Iface:"cali3148282adac", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:21, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:24, Iface:"cali3d68ded7758", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:26, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.976 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:39, Iface:"cali1f715c523aa", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:36, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.976 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:32, Iface:"cali52c6ce56514", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:34, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.976 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:23, Iface:"cali3148282adac", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:20, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:12, Iface:"cali9135c56a46c", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:14, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:33, Iface:"cali52c6ce56514", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:35, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:38, Iface:"cali1f715c523aa", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:37, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:28, Iface:"calicf78b17719a", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:30, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.979 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:13, Iface:"cali9135c56a46c", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:15, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.979 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:40, Iface:"cali0ecd3ac8233", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:42, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.979 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:41, Iface:"cali0ecd3ac8233", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:43, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
...

These errors can be resolved by deleting the file /sys/fs/bpf/tc/globals/cali_v4_ct3. But, when I tried to dump the conntrack table, another error occurred: ErrIterationFinished during the first iteration.

// PinnedMap &{MapParameters:{PinDir: Type:hash KeySize:16 ValueSize:88 MaxEntries:100000 Name:cali_v4_ct Flags:1 Version:3 UpdatedByBPF:true} fdLoaded:false fd:0 oldfd:0 perCPU:false oldSize:0 UpgradeFn:0x1a07d20 GetMapParams:0x2aa3460 
func (b *PinnedMap) Iter(f IterCallback) error {
...
    // valueSize: 88 KeySize: 16 MaxEntries: 100000 MapFD: 8
    it, err := NewIterator(b.MapFD(), b.KeySize, valueSize, b.MaxEntries)
....
	var action IteratorAction
	for {
		k, v, err := it.Next()
...
		if err != nil {
			if err == ErrIterationFinished {
				return nil
			}
			return errors.Errorf("iterating the map failed: %s", err)
		}
...
}
@ioworker0
Copy link
Contributor Author

cc @tomastigera

@ioworker0 ioworker0 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 5, 2025
@tomastigera
Copy link
Contributor

What was the resolution of this?

@ioworker0 ioworker0 reopened this Jan 7, 2025
@ioworker0
Copy link
Contributor Author

What was the resolution of this?

Just closed for vacation ;p

We haven't fixed this bug yet, but found some clues that might be helpful.

calico/felix/bpf-gpl/conntrack_types.h

Lines 156 to 163 in 342d8ec

#ifdef IPVER6
CALI_MAP_NAMED(cali_v6_ct, cali_ct, 3,
#else
CALI_MAP_NAMED(cali_v4_ct, cali_ct, 3,
#endif
BPF_MAP_TYPE_HASH,
struct calico_ct_key, struct calico_ct_value,
512000, BPF_F_NO_PREALLOC)

calico/felix/config/config_params.go

Lines 199 to 203 in 342d8ec

BPFMapSizeNATAffinity int `config:"int;65536;non-zero"`
BPFMapSizeRoute int `config:"int;262144;non-zero"`
BPFMapSizeConntrack int `config:"int;512000;non-zero"`
BPFMapSizePerCPUConntrack int `config:"int;0"`
BPFMapSizeConntrackCleanupQueue int `config:"int;100000;non-zero"`

calico/felix/bpf/conntrack/v3/map.go

Lines 34 to 36 in 342d8ec

const KeySize = 16
const ValueSize = 88
const MaxEntries = 512000

calico/felix/bpf/conntrack/v2/map.go

Lines 34 to 36 in 342d8ec

const KeySize = 16
const ValueSize = 64
const MaxEntries = 512000

As long as we modify these values(512000) at the same time, and rebuild the image, then everything will be working as expected.

@ioworker0
Copy link
Contributor Author

CC @tomastigera @fasaxc

@tomastigera tomastigera added kind/bug area/bpf eBPF Dataplane issues labels Jan 7, 2025
@tomastigera
Copy link
Contributor

The problem is most likely in https://github.com/projectcalico/calico/blob/master/felix/cmd/calico-bpf/commands/conntrack.go where we do not adjust the default value to the value from the actual map. That is a bug. I think we need to read the size and then call https://github.com/projectcalico/calico/blob/master/felix/bpf/libbpf/libbpf.go#L87

@ioworker0
Copy link
Contributor Author

The problem is most likely in https://github.com/projectcalico/calico/blob/master/felix/cmd/calico-bpf/commands/conntrack.go where we do not adjust the default value to the value from the actual map. That is a bug. I think we need to read the size and then call https://github.com/projectcalico/calico/blob/master/felix/bpf/libbpf/libbpf.go#L87

Good catch! Thanks a lot for your time!

Unfortunately, it seems that there is more than one problem :(

  1. The default value should be adjusted to match the actual value when dumping the conntrack map.
  2. We cannot reuse the pinned map after modifying the global Felix configuration and restarting the calico-node.
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'

calico/felix/bpf-gpl/include/libbpf/src/libbpf.c

Lines 5269 to 5275 in 638464f

retry:
if (map->pin_path) {
err = bpf_object__reuse_map(map);
if (err) {
pr_warn("map '%s': error reusing pinned map\n",
map->name);
goto err_out;

I‘m not sure whether the second problem still exists in the master, but everything is working as expected after the following changes in v3.27.3.

calico/felix/bpf-gpl/conntrack_types.h

Lines 156 to 163 in 342d8ec

#ifdef IPVER6
CALI_MAP_NAMED(cali_v6_ct, cali_ct, 3,
#else
CALI_MAP_NAMED(cali_v4_ct, cali_ct, 3,
#endif
BPF_MAP_TYPE_HASH,
struct calico_ct_key, struct calico_ct_value,
512000, BPF_F_NO_PREALLOC)

The value 512000 in this macro needs to be changed to one that aligns with the global Felix configuration.

@tomastigera
Copy link
Contributor

tomastigera commented Jan 8, 2025 •
edited
Loading

If you would like to submit a fix for (1), that would be fantastic!

As for (2), that is weird, we specifically test that here https://github.com/projectcalico/calico/blob/master/felix/fv/bpf_map_resize_test.go Note that there is a process of creating a new map, copying over the old entries and then removing the old map. Also I am pretty sure that we adjust the map constant at loadtime through libbpf 🤔 @sridhartigera

@sridhartigera
Copy link
Member

There was a fix related to this in 3.29 regarding (2). When attaching BPF programs, the new size is picked up. #9117

@ioworker0
Copy link
Contributor Author

If you would like to submit a fix for (1), that would be fantastic!

Thanks for reaching out!

Yep, let me give it a try ~

@ioworker0
Copy link
Contributor Author

There was a fix related to this in 3.29 regarding (2). When attaching BPF programs, the new size is picked up. #9117

Thanks for sharing and referencing. It helps me a lot!

@ioworker0 ioworker0 linked a pull request Jan 13, 2025 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/bpf eBPF Dataplane issues kind/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[BPF] Automatically adjust to actual max-entries when dumping conntra… ioworker0/calico
3 participants
@ioworker0 @tomastigera @sridhartigera