Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zebra crashes with segv working with netns-based VRFs #15164

Closed
odivlad opened this issue Jan 16, 2024 · 9 comments
Closed

Zebra crashes with segv working with netns-based VRFs #15164

odivlad opened this issue Jan 16, 2024 · 9 comments
Assignees
@pguibert6WIND

Comments

@odivlad
Copy link
Contributor

odivlad commented Jan 16, 2024

I run FRR with next /etc/frr/daemons config:

bgpd=yes
vtysh_enable=yes
zebra_options="  -A 127.0.0.1 -s 90000000 --vrfwnetns"
bgpd_options="   -A 127.0.0.1"

Sometimes when adding or deleting netns-based VRF zebra daemon crashes with SIGSEGV.

When create with these commands:

ip netns add dxpeer-A0869961
ip link add dxpeer-A0869961 type veth peer name eth0 netns dxpeer-A0869961
ip netns exec dxpeer-A0869961 ip link set eth0 up
ip link set dxpeer-A0869961 up

sometimes daemon crashes with next trace.

zebra[30200]: [GDAET-J7Q7G] NS notify : created VRF dxpeer-A0869961 NS /run/netns/dxpeer-A0869961
ZEBRA[30200]: Received signal 11 at 1705439706 (si_addr 0x14fcaab94, PC 0x475388); aborting...
ZEBRA[30200]: /usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7fc7f09a6a44]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7fc7f09a6c5c]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(+0xc2662) [0x7fc7f09cf662]
ZEBRA[30200]: /usr/lib64/libpthread.so.0(+0xf630) [0x7fc7ef64e630]
ZEBRA[30200]: /usr/lib/frr/zebra() [0x475388]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7fc7f09e9630]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7fc7f09df53b]
ZEBRA[30200]: /usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7fc7f099f8f0]
ZEBRA[30200]: /usr/lib/frr/zebra(main+0x378) [0x41e868]
ZEBRA[30200]: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc7ef293555]
ZEBRA[30200]: /usr/lib/frr/zebra() [0x41f4e8]
ZEBRA[30200]: in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()

When delete with these commands:

ip netns exec dxvif-13C916E0 ip link set eth0 down
ip netns delete dxvif-13C916E0
vtysh -c 'conf t
no vrf dxvif-13C916E0
end
write memory'

sometimes I get this trace:

ZEBRA[10400]: Received signal 11 at 1705438022 (si_addr 0x100000041, PC 0x475388); aborting...
ZEBRA[10400]: /usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7fae5d743a44]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7fae5d743c5c]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(+0xc2662) [0x7fae5d76c662]
ZEBRA[10400]: /usr/lib64/libpthread.so.0(+0xf630) [0x7fae5c3eb630]
ZEBRA[10400]: /usr/lib/frr/zebra() [0x475388]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7fae5d786630]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7fae5d77c53b]
ZEBRA[10400]: /usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7fae5d73c8f0]
ZEBRA[10400]: /usr/lib/frr/zebra(main+0x378) [0x41e868]
ZEBRA[10400]: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fae5c030555]
ZEBRA[10400]: /usr/lib/frr/zebra() [0x41f4e8]
watchfrr[17333]: [HD38Q-0HBRT][EC 268435457] zebra state -> down : read returned EOF

Strace output right before and during crash:

[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base={{len=1284, type=RTM_NEWLINK, flags=0, seq=0, pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_ETHER, ifi_index=if_nametoindex("dxpeer-BFDCD441"), ifi_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST|IFF_LOWER_UP, ifi_change=0}, [{{nla_len=20, nla_type=IFLA_IFNAME}, "dxpeer-BFDCD441"}, {{nla_len=8, nla_type=IFLA_TXQLEN}, 1000}, {{nla_len=5, nla_type=IFLA_OPERSTATE}, 6}, {{nla_len=5, nla_type=IFLA_LINKMODE}, 0}, {{nla_len=8, nla_type=IFLA_MTU}, 1500}, {{nla_len=8, nla_type=IFLA_GROUP}, 0}, {{nla_len=8, nla_type=IFLA_PROMISCUITY}, 0}, {{nla_len=8, nla_type=IFLA_NUM_TX_QUEUES}, 1}, {{nla_len=8, nla_type=IFLA_GSO_MAX_SEGS}, 65535}, {{nla_len=8, nla_type=IFLA_GSO_MAX_SIZE}, 65536}, {{nla_len=8, nla_type=IFLA_NUM_RX_QUEUES}, 1}, {{nla_len=5, nla_type=IFLA_CARRIER}, 1}, {{nla_len=12, nla_type=IFLA_QDISC}, "noqueue"}, {{nla_len=8, nla_type=IFLA_CARRIER_CHANGES}, 2}, {{nla_len=5, nla_type=IFLA_PROTO_DOWN}, 0}, {{nla_len=36, nla_type=IFLA_MAP}, {mem_start=0, mem_end=0, base_addr=0, irq=0, dma=0, port=0}}, {{nla_len=10, nla_type=IFLA_ADDRESS}, "\xd6\x79\xe5\x6c\xda\x71"}, {{nla_len=10, nla_type=IFLA_BROADCAST}, "\xff\xff\xff\xff\xff\xff"}, {{nla_len=196, nla_type=IFLA_STATS64}, {rx_packets=0, tx_packets=0, rx_bytes=0, tx_bytes=0, rx_errors=0, tx_errors=0, rx_dropped=0, tx_dropped=0, multicast=0, collisions=0, rx_length_errors=0, rx_over_errors=0, rx_crc_errors=0, rx_frame_errors=0, rx_fifo_errors=0, rx_missed_errors=0, tx_aborted_errors=0, tx_carrier_errors=0, tx_fifo_errors=0, tx_heartbeat_errors=0, tx_window_errors=0, rx_compressed=0, tx_compressed=0, rx_nohandler=0}}, {{nla_len=100, nla_type=IFLA_STATS}, {rx_packets=0, tx_packets=0, rx_bytes=0, tx_bytes=0, rx_errors=0, tx_errors=0, rx_dropped=0, tx_dropped=0, multicast=0, collisions=0, rx_length_errors=0, rx_over_errors=0, rx_crc_errors=0, rx_frame_errors=0, rx_fifo_errors=0, rx_missed_errors=0, tx_aborted_errors=0, tx_carrier_errors=0, tx_fifo_errors=0, tx_heartbeat_errors=0, tx_window_errors=0, rx_compressed=0, tx_compressed=0, rx_nohandler=0}}, {{nla_len=16, nla_type=IFLA_LINKINFO}, {{nla_len=9, nla_type=IFLA_INFO_KIND}, "veth"}}, {{nla_len=8, nla_type=IFLA_LINK_NETNSID}, 4}, {{nla_len=8, nla_type=IFLA_LINK}, 8}, {{nla_len=728, nla_type=IFLA_AF_SPEC}, [{{nla_len=120, nla_type=AF_INET}, {{nla_len=116, nla_type=IFLA_INET_CONF}, [[IPV4_DEVCONF_FORWARDING-1] = 0, [IPV4_DEVCONF_MC_FORWARDING-1] = 0, [IPV4_DEVCONF_PROXY_ARP-1] = 0, [IPV4_DEVCONF_ACCEPT_REDIRECTS-1] = 0, [IPV4_DEVCONF_SECURE_REDIRECTS-1] = 1, [IPV4_DEVCONF_SEND_REDIRECTS-1] = 0, [IPV4_DEVCONF_SHARED_MEDIA-1] = 1, [IPV4_DEVCONF_RP_FILTER-1] = 1, [IPV4_DEVCONF_ACCEPT_SOURCE_ROUTE-1] = 0, [IPV4_DEVCONF_BOOTP_RELAY-1] = 0, [IPV4_DEVCONF_LOG_MARTIANS-1] = 0, [IPV4_DEVCONF_TAG-1] = 0, [IPV4_DEVCONF_ARPFILTER-1] = 0, [IPV4_DEVCONF_MEDIUM_ID-1] = 0, [IPV4_DEVCONF_NOXFRM-1] = 0, [IPV4_DEVCONF_NOPOLICY-1] = 0, [IPV4_DEVCONF_FORCE_IGMP_VERSION-1] = 0, [IPV4_DEVCONF_ARP_ANNOUNCE-1] = 0, [IPV4_DEVCONF_ARP_IGNORE-1] = 0, [IPV4_DEVCONF_PROMOTE_SECONDARIES-1] = 1, [IPV4_DEVCONF_ARP_ACCEPT-1] = 0, [IPV4_DEVCONF_ARP_NOTIFY-1] = 0, [IPV4_DEVCONF_ACCEPT_LOCAL-1] = 0, [IPV4_DEVCONF_SRC_VMARK-1] = 0, [IPV4_DEVCONF_PROXY_ARP_PVLAN-1] = 0, [IPV4_DEVCONF_ROUTE_LOCALNET-1] = 0, [IPV4_DEVCONF_IGMPV2_UNSOLICITED_REPORT_INTERVAL-1] = 10000, [IPV4_DEVCONF_IGMPV3_UNSOLICITED_REPORT_INTERVAL-1] = 1000]}}, {{nla_len=604, nla_type=AF_INET6}, [{{nla_len=8, nla_type=IFLA_INET6_FLAGS}, IF_READY}, {{nla_len=20, nla_type=IFLA_INET6_CACHEINFO}, {max_reasm_len=65535, tstamp=1342870334, reachable_time=22610, retrans_time=1000}}, {{nla_len=192, nla_type=IFLA_INET6_CONF}, [[DEVCONF_FORWARDING] = 0, [DEVCONF_HOPLIMIT] = 64, [DEVCONF_MTU6] = 1500, [DEVCONF_ACCEPT_RA] = 1, [DEVCONF_ACCEPT_REDIRECTS] = 0, [DEVCONF_AUTOCONF] = 1, [DEVCONF_DAD_TRANSMITS] = 1, [DEVCONF_RTR_SOLICITS] = 3, [DEVCONF_RTR_SOLICIT_INTERVAL] = 4000, [DEVCONF_RTR_SOLICIT_DELAY] = 1000, [DEVCONF_USE_TEMPADDR] = 0, [DEVCONF_TEMP_VALID_LFT] = 604800, [DEVCONF_TEMP_PREFERED_LFT] = 86400, [DEVCONF_REGEN_MAX_RETRY] = 3, [DEVCONF_MAX_DESYNC_FACTOR] = 600, [DEVCONF_MAX_ADDRESSES] = 16, [DEVCONF_FORCE_MLD_VERSION] = 0, [DEVCONF_ACCEPT_RA_DEFRTR] = 1, [DEVCONF_ACCEPT_RA_PINFO] = 1, [DEVCONF_ACCEPT_RA_RTR_PREF] = 1, [DEVCONF_RTR_PROBE_INTERVAL] = 60000, [DEVCONF_ACCEPT_RA_RT_INFO_MAX_PLEN] = 0, [DEVCONF_PROXY_NDP] = 0, [DEVCONF_OPTIMISTIC_DAD] = 0, [DEVCONF_ACCEPT_SOURCE_ROUTE] = 0, [DEVCONF_MC_FORWARDING] = 0, [DEVCONF_DISABLE_IPV6] = 1, [DEVCONF_ACCEPT_DAD] = 1, [DEVCONF_FORCE_TLLAO] = 0, [DEVCONF_NDISC_NOTIFY] = 0, [DEVCONF_MLDV1_UNSOLICITED_REPORT_INTERVAL] = 10000, [DEVCONF_MLDV2_UNSOLICITED_REPORT_INTERVAL] = 1000, [DEVCONF_SUPPRESS_FRAG_NDISC] = 0, [DEVCONF_ACCEPT_RA_FROM_LOCAL] = 0, [DEVCONF_USE_OPTIMISTIC] = 0, [DEVCONF_ACCEPT_RA_MTU] = 0, [DEVCONF_STABLE_SECRET] = 0, [DEVCONF_USE_OIF_ADDRS_ONLY] = 0, [DEVCONF_ACCEPT_RA_MIN_HOP_LIMIT] = 0, [DEVCONF_IGNORE_ROUTES_WITH_LINKDOWN] = 0, [DEVCONF_DROP_UNICAST_IN_L2_MULTICAST] = 0, [DEVCONF_DROP_UNSOLICITED_NA] = 0, [DEVCONF_KEEP_ADDR_ON_DOWN] = 0, [DEVCONF_RTR_SOLICIT_MAX_INTERVAL] = 0, [DEVCONF_SEG6_ENABLED] = 0, [DEVCONF_SEG6_REQUIRE_HMAC] = 0, [DEVCONF_ENHANCED_DAD] = 1]}, {{nla_len=300, nla_type=IFLA_INET6_STATS}, [[IPSTATS_MIB_NUM] = 37, [IPSTATS_MIB_INPKTS] = 0, [IPSTATS_MIB_INOCTETS] = 0, [IPSTATS_MIB_INDELIVERS] = 0, [IPSTATS_MIB_OUTFORWDATAGRAMS] = 0, [IPSTATS_MIB_OUTPKTS] = 0, [IPSTATS_MIB_OUTOCTETS] = 0, [IPSTATS_MIB_INHDRERRORS] = 0, [IPSTATS_MIB_INTOOBIGERRORS] = 0, [IPSTATS_MIB_INNOROUTES] = 0, [IPSTATS_MIB_INADDRERRORS] = 0, [IPSTATS_MIB_INUNKNOWNPROTOS] = 0, [IPSTATS_MIB_INTRUNCATEDPKTS] = 0, [IPSTATS_MIB_INDISCARDS] = 0, [IPSTATS_MIB_OUTDISCARDS] = 0, [IPSTATS_MIB_OUTNOROUTES] = 0, [IPSTATS_MIB_REASMTIMEOUT] = 0, [IPSTATS_MIB_REASMREQDS] = 0, [IPSTATS_MIB_REASMOKS] = 0, [IPSTATS_MIB_REASMFAILS] = 0, [IPSTATS_MIB_FRAGOKS] = 0, [IPSTATS_MIB_FRAGFAILS] = 0, [IPSTATS_MIB_FRAGCREATES] = 0, [IPSTATS_MIB_INMCASTPKTS] = 0, [IPSTATS_MIB_OUTMCASTPKTS] = 0, [IPSTATS_MIB_INBCASTPKTS] = 0, [IPSTATS_MIB_OUTBCASTPKTS] = 0, [IPSTATS_MIB_INMCASTOCTETS] = 0, [IPSTATS_MIB_OUTMCASTOCTETS] = 0, [IPSTATS_MIB_INBCASTOCTETS] = 0, [IPSTATS_MIB_OUTBCASTOCTETS] = 0, [IPSTATS_MIB_CSUMERRORS] = 0, [IPSTATS_MIB_NOECTPKTS] = 0, [IPSTATS_MIB_ECT1PKTS] = 0, [IPSTATS_MIB_ECT0PKTS] = 0, [IPSTATS_MIB_CEPKTS] = 0, [36 /* IPSTATS_MIB_??? */] = 0]}, {{nla_len=52, nla_type=IFLA_INET6_ICMP6STATS}, [[ICMP6_MIB_NUM] = 6, [ICMP6_MIB_INMSGS] = 0, [ICMP6_MIB_INERRORS] = 0, [ICMP6_MIB_OUTMSGS] = 0, [ICMP6_MIB_OUTERRORS] = 0, [ICMP6_MIB_CSUMERRORS] = 0]}, {{nla_len=20, nla_type=IFLA_INET6_TOKEN}, ::}, {{nla_len=5, nla_type=IFLA_INET6_ADDR_GEN_MODE}, IN6_ADDR_GEN_MODE_EUI64}]}]}]}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1284
[pid  4888] read(54,  <unfinished ...>
[pid  4885] read(49,  <unfinished ...>
[pid  4876] recvmsg(12,  <unfinished ...>
[pid  4888] <... read resumed>0x7f3f721f0c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4885] <... read resumed>0x7f3f729f1c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000004}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=RTM_NEWNEIGH, flags=0, seq=0, pid=0}, {ndm_family=AF_INET, ndm_ifindex=if_nametoindex("cluster"), ndm_state=NUD_REACHABLE, ndm_flags=0, ndm_type=RTN_UNICAST}, [{{nla_len=8, nla_type=NDA_DST}, 10.1.1.21}, {{nla_len=10, nla_type=NDA_LLADDR}, 82.84.0.73}, {{nla_len=8, nla_type=NDA_PROBES}, 1}, {{nla_len=20, nla_type=NDA_CACHEINFO}, {ndm_confirmed=451, ndm_used=451, ndm_updated=0, ndm_refcnt=2}}]}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 76
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4888] ppoll([{fd=53, events=POLLIN|POLLOUT}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] recvmsg(12,  <unfinished ...>
[pid  4888] <... ppoll resumed>)        = 1 ([{fd=53, revents=POLLOUT}])
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] <... recvmsg resumed>{msg_namelen=12}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4888] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4885] ppoll([{fd=37, events=POLLIN|POLLOUT}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] write(8, "\1", 1 <unfinished ...>
[pid  4888] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4885] <... ppoll resumed>)        = 1 ([{fd=37, revents=POLLOUT}])
[pid  4876] <... write resumed>)        = 1
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4876] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28615}, ...}) = 0
[pid  4885] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4876] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295500}, ...}) = 0
[pid  4888] writev(53, [{iov_base="\0T\376\6\0\0\0\0\0\5dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20\3\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0\0T\376\6\0\0\0\0\0\4dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0", iov_len=168}], 1 <unfinished ...>
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4888] <... writev resumed>)       = 168
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33073}, ...}) = 0
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000400}, msg_namelen=12, msg_iov=[{iov_base={{len=116, type=0x18 /* NLMSG_??? */, flags=0, seq=0, pid=0}, "\x0a\x08\x00\x00\xff\x03\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xff\x00\x00\x00\x14\x00\x01\x00\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x04\x00\x08\x00\x00\x00\x08\x00\x06\x00\x00\x01\x00\x00\x24\x00\x0c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x14\x00\x00\x00\x00\x00"}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 116
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] writev(37, [{iov_base="\0T\376\6\0\0\0\0\0\5dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20\3\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0\0T\376\6\0\0\0\0\0\4dxpeer-BFDCD441\0\0\0\0\0\0\0\t)\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\326y\345l\332q\0", iov_len=168}], 1 <unfinished ...>
[pid  4876] write(8, "\1", 1 <unfinished ...>
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28680}, ...}) = 0
[pid  4885] <... writev resumed>)       = 168
[pid  4876] <... write resumed>)        = 1
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000400}, msg_namelen=12, msg_iov=[{iov_base={{len=116, type=0x18 /* NLMSG_??? */, flags=0, seq=0, pid=0}, "\x0a\x40\x00\x00\xfe\x02\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xfe\x00\x00\x00\x14\x00\x01\x00\xfe\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x04\x00\x08\x00\x00\x00\x08\x00\x06\x00\x00\x01\x00\x00\x24\x00\x0c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x14\x00\x00\x00\x00\x00"}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 116
[pid  4888] ppoll([{fd=53, events=POLLIN}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33132}, ...}) = 0
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base={{len=1276, type=0x10 /* NLMSG_??? */, flags=0, seq=0, pid=0}, "\x00\x00\x01\x00\x08\x00\x00\x00\x43\x10\x01\x00\x00\x00\x00\x00\x09\x00\x03\x00\x65\x74\x68\x30\x00\x00\x00\x00\x08\x00\x0d\x00\xe8\x03\x00\x00\x05\x00\x10\x00\x06\x00\x00\x00\x05\x00\x11\x00\x00\x00\x00\x00\x08\x00\x04\x00\xdc\x05\x00\x00\x08\x00\x1b\x00\x00\x00\x00\x00\x08\x00\x1e\x00\x00\x00\x00\x00\x08\x00\x1f\x00\x01\x00\x00\x00\x08\x00\x28\x00\xff\xff\x00\x00\x08\x00\x29\x00\x00\x00\x01\x00\x08\x00\x20\x00\x01\x00\x00\x00\x05\x00\x21\x00\x01\x00\x00\x00\x0c\x00\x06\x00\x6e\x6f\x71\x75\x65\x75\x65\x00\x08\x00\x23\x00\x02\x00\x00\x00\x05\x00\x27\x00\x00\x00\x00\x00\x24\x00\x0e\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0a\x00\x01\x00\xfe\xcf\x39\x0e\xcb\x94\x00\x00\x0a\x00\x02\x00\xff\xff\xff\xff\xff\xff\x00\x00\xc4\x00\x17\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x64\x00\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x12\x00\x09\x00\x01\x00\x76\x65\x74\x68\x00\x00\x00\x00\x08\x00\x25\x00\x00\x00\x00\x00\x08\x00\x05\x00\x29\x09\x00\x00\xd8\x02\x1a\x00\x78\x00\x02\x00\x74\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x27\x00\x00\xe8\x03\x00\x00\x5c\x02\x0a\x00\x08\x00\x01\x00\x00\x00\x00\x80\x14\x00\x05\x00\xff\xff\x00\x00\x3e\x93\x0a\x50\x0d\x75\x00\x00\xe8\x03\x00\x00\xc0\x00\x02\x00\x00\x00\x00\x00\x40\x00\x00\x00\xdc\x05\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x03\x00\x00\x00\xa0\x0f\x00\x00\xe8\x03\x00\x00\x00\x00\x00\x00\x80\x3a\x09\x00\x80\x51\x01\x00\x03\x00\x00\x00\x58\x02\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x60\xea\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x27\x00\x00\xe8\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x2c\x01\x03\x00\x25\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"...}, iov_len=32768}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1276
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] write(50, "\1", 1 <unfinished ...>
[pid  4885] ppoll([{fd=37, events=POLLIN}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] <... write resumed>)        = 1
[pid  4885] <... ppoll resumed>)        = 1 ([{fd=49, revents=POLLIN}])
[pid  4876] write(55, "\1", 1 <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4876] <... write resumed>)        = 1
[pid  4888] <... ppoll resumed>)        = 1 ([{fd=54, revents=POLLIN}])
[pid  4885] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4876] recvmsg(33,  <unfinished ...>
[pid  4888] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4885] read(49,  <unfinished ...>
[pid  4876] <... recvmsg resumed>{msg_namelen=12}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4888] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4885] <... read resumed>"\1", 64) = 1
[pid  4876] write(8, "\1", 1 <unfinished ...>
[pid  4888] read(54,  <unfinished ...>
[pid  4885] read(49,  <unfinished ...>
[pid  4876] <... write resumed>)        = 1
[pid  4888] <... read resumed>"\1", 64) = 1
[pid  4885] <... read resumed>0x7f3f729f1c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4876] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] read(54,  <unfinished ...>
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295744}, ...}) = 0
[pid  4888] <... read resumed>0x7f3f721f0c30, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4885] ppoll([{fd=37, events=POLLIN|POLLOUT}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 TERM],  <unfinished ...>
[pid  4888] ppoll([{fd=53, events=POLLIN|POLLOUT}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4885] <... ppoll resumed>)        = 1 ([{fd=37, revents=POLLOUT}])
[pid  4888] <... ppoll resumed>)        = 1 ([{fd=53, revents=POLLOUT}])
[pid  4885] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4876] <... rt_sigprocmask resumed>[], 8) = 0
[pid  4888] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1],  <unfinished ...>
[pid  4885] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4888] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4876] ppoll([{fd=12, events=POLLIN}, {fd=15, events=POLLIN}, {fd=43, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=58, events=POLLIN}, {fd=18, events=POLLIN}, {fd=23, events=POLLIN}, {fd=28, events=POLLIN}, {fd=33, events=POLLIN}, {fd=7, events=POLLIN}], 11, {tv_sec=0, tv_nsec=0}, [], 8 <unfinished ...>
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28807}, ...}) = 0
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33261}, ...}) = 0
[pid  4888] writev(53, [{iov_base="\0T\376\6\0\0\0\2\0\4eth0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\376\3179\16\313\224\0", iov_len=84}], 1 <unfinished ...>
[pid  4876] <... ppoll resumed>)        = 1 ([{fd=7, revents=POLLIN}], left {tv_sec=0, tv_nsec=0})
[pid  4888] <... writev resumed>)       = 84
[pid  4885] writev(37, [{iov_base="\0T\376\6\0\0\0\2\0\4eth0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\5\0\0\0\0\0\1\20C\0\2\0\0\0\0\0\0\0\0\0\0\5\334\0\0\5\334\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\376\3179\16\313\224\0", iov_len=84}], 1 <unfinished ...>
[pid  4876] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid  4885] <... writev resumed>)       = 84
[pid  4888] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4876] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid  4888] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=4447}, ru_stime={tv_sec=0, tv_usec=28836}, ...}) = 0
[pid  4876] read(7,  <unfinished ...>
[pid  4888] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] <... read resumed>"\1\1\1\1", 64) = 4
[pid  4888] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4885] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4888] ppoll([{fd=53, events=POLLIN}, {fd=56, events=POLLIN}, {fd=54, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] read(7,  <unfinished ...>
[pid  4885] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=9704}, ru_stime={tv_sec=0, tv_usec=33293}, ...}) = 0
[pid  4876] <... read resumed>0x7ffe5328a300, 64) = -1 EAGAIN (Resource temporarily unavailable)
[pid  4885] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid  4876] getrusage(RUSAGE_THREAD,  <unfinished ...>
[pid  4885] <... rt_sigprocmask resumed>~[KILL STOP RTMIN RT_1], 8) = 0
[pid  4876] <... getrusage resumed>{ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295828}, ...}) = 0
[pid  4885] ppoll([{fd=37, events=POLLIN}, {fd=51, events=POLLIN}, {fd=49, events=POLLIN}], 3, NULL, ~[KILL STOP RTMIN RT_1], 8 <unfinished ...>
[pid  4876] getrusage(RUSAGE_THREAD, {ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=295873}, ...}) = 0
[pid  4876] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 TERM], [], 8) = 0
[pid  4876] ppoll([{fd=12, events=POLLIN}, {fd=15, events=POLLIN}, {fd=43, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=58, events=POLLIN}, {fd=18, events=POLLIN}, {fd=23, events=POLLIN}, {fd=28, events=POLLIN}, {fd=33, events=POLLIN}, {fd=7, events=POLLIN}], 11, {tv_sec=0, tv_nsec=5000000}, [], 8) = 0 (Timeout)
[pid  4876] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  4876] rt_sigprocmask(SIG_BLOCK, [HUP INT USR1 TERM], [], 8) = 0
[pid  4876] ppoll([{fd=12, events=POLLIN}, {fd=15, events=POLLIN}, {fd=43, events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=58, events=POLLIN}, {fd=18, events=POLLIN}, {fd=23, events=POLLIN}, {fd=28, events=POLLIN}, {fd=33, events=POLLIN}, {fd=7, events=POLLIN}], 11, {tv_sec=0, tv_nsec=0}, [], 8) = 0 (Timeout)
[pid  4876] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  4876] getrusage(RUSAGE_THREAD, {ru_utime={tv_sec=0, tv_usec=259937}, ru_stime={tv_sec=0, tv_usec=296070}, ...}) = 0
[pid  4876] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x41} ---
[pid  4876] rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f3f76a29630}, NULL, 8) = 0
[pid  4876] rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f3f76a29630}, NULL, 8) = 0
[pid  4876] rt_sigprocmask(SIG_UNBLOCK, [ALRM], NULL, 8) = 0
[pid  4876] alarm(1)                    = 0
[pid  4876] openat(3, "crashlog", O_WRONLY|O_CREAT|O_APPEND, 0600) = 59
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="Received signal 11 at 1705442216 (si_addr 0x41, PC 0x475388); aborting...", iov_len=73}, {iov_base="\n", iov_len=1}], 4) = 81
[pid  4876] socket(AF_UNIX, SOCK_DGRAM, 0) = 60
[pid  4876] connect(60, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="Received signal 11 at 1705442216 (si_addr 0x41, PC 0x475388); aborting...", iov_len=73}], 2) = 89
[pid  4876] futex(0x7f3f76a17cc0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  4876] futex(0x7f3f76e4b190, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7f3f77d81a44]", iov_len=68}, {iov_base="\n", iov_len=1}], 4) = 76
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(zlog_backtrace_sigsafe+0x74) [0x7f3f77d81a44]", iov_len=68}], 2) = 84
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7f3f77d81c5c]", iov_len=57}, {iov_base="\n", iov_len=1}], 4) = 65
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(zlog_signal+0xec) [0x7f3f77d81c5c]", iov_len=57}], 2) = 73
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(+0xc2662) [0x7f3f77daa662]", iov_len=49}, {iov_base="\n", iov_len=1}], 4) = 57
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(+0xc2662) [0x7f3f77daa662]", iov_len=49}], 2) = 65
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libpthread.so.0(+0xf630) [0x7f3f76a29630]", iov_len=52}, {iov_base="\n", iov_len=1}], 4) = 60
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libpthread.so.0(+0xf630) [0x7f3f76a29630]", iov_len=52}], 2) = 68
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib/frr/zebra() [0x475388]", iov_len=31}, {iov_base="\n", iov_len=1}], 4) = 39
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib/frr/zebra() [0x475388]", iov_len=31}], 2) = 47
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7f3f77dc4630]", iov_len=60}, {iov_base="\n", iov_len=1}], 4) = 68
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(work_queue_run+0xd0) [0x7f3f77dc4630]", iov_len=60}], 2) = 76
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7f3f77dba53b]", iov_len=57}, {iov_base="\n", iov_len=1}], 4) = 65
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(thread_call+0x7b) [0x7f3f77dba53b]", iov_len=57}], 2) = 73
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7f3f77d7a8f0]", iov_len=53}, {iov_base="\n", iov_len=1}], 4) = 61
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libfrr.so.0(frr_run+0xe0) [0x7f3f77d7a8f0]", iov_len=53}], 2) = 69
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib/frr/zebra(main+0x378) [0x41e868]", iov_len=41}, {iov_base="\n", iov_len=1}], 4) = 49
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib/frr/zebra(main+0x378) [0x41e868]", iov_len=41}], 2) = 57
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3f7666e555]", iov_len=61}, {iov_base="\n", iov_len=1}], 4) = 69
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3f7666e555]", iov_len=61}], 2) = 77
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="/usr/lib/frr/zebra() [0x41f4e8]", iov_len=31}, {iov_base="\n", iov_len=1}], 4) = 39
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="/usr/lib/frr/zebra() [0x41f4e8]", iov_len=31}], 2) = 47
[pid  4876] writev(59, [{iov_base="", iov_len=0}, {iov_base="ZEBRA: ", iov_len=7}, {iov_base="in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()", iov_len=81}, {iov_base="\n", iov_len=1}], 4) = 89
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()", iov_len=81}], 2) = 97
[pid  4876] writev(60, [{iov_base="<2>ZEBRA[4876]: ", iov_len=16}, {iov_base="in thread work_queue_run scheduled from lib/workqueue.c:140 work_queue_schedule()", iov_len=81}], 2) = 97
[pid  4876] write(2, "core_handler: showing active allocations in memory group libfrr\n", 64) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Buffer                        :      4 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Host config                   :      3 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Tokens                :   4144 *         72\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Token Text            :   2967 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Token Help            :   2967 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Command Argument Name         :   1043 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  RCU thread                    :      4 *        128\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  FRR POSIX Thread              :      8 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  POSIX sync primitives         :      8 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Graph                         :     25 *          8\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Graph Node                    :   4848 *         32\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hash                          :    548 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hash Bucket                   :    947 *         32\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hash Index                    :    274 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Hook entry                    :      8 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Interface                     :     55 *        272\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Connected                     :     11 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Link List                     :    313 *         40\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Link Node                     :    350 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Temporary memory              :      4 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop                       :     21 *        136\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  NetNS Context                 :     11 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  NetNS Name                    :      5 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Northbound Node               :    655 *       1192\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Northbound Configuration      :      2 *         16\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Northbound Configuration Entry:     11 *       1032\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Prefix                        :     11 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Privilege information         :      4 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route map                     :      4 *        120\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route map name                :      4 *         23\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route map index               :      2 *        152\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Stream                        :      4 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Stream FIFO                   :      5 *         64\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route table                   :    119 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route node                    :    198 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread                        :     41 *        152\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread master                 :     20 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread Poll Info              :     10 *       8192\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Thread stats                  :     26 *         88\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Typed-hash bucket             :     43 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Typed-heap array              :      1 *        576\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Vector                        :   9752 *         16\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Vector index                  :   9752 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  VRF                           :      5 *        216\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  VRF bit-map                   :     11 *          8\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  VTY                           :      2 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Work queue                    :      2 * (variably sized)\n", 83) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Work queue item               :      1 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Work queue name string        :      1 *         22\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  YANG module                   :      7 *         48\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  log thread-local buffer       :      5 *      10272\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group logging subsystem\n", 75) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  syslog target                 :      1 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group Label Manager\n", 71) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group Table Manager\n", 71) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: showing active allocations in memory group zebra\n", 63) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra Interface Information   :     55 *        472\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra Netlink buffers         :      1 *     131072\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Opaque Data                   :      3 *          8\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra DPlane Provider         :      1 *        232\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop Group Entry           :     20 *        112\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop Group Connected       :     19 *         40\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra Name Space              :      5 *        320\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  PTM BFD process registration table.:      1 *         32\n", 82) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Route Entry                   :     43 *         96\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  RIB destination               :     61 *         88\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  RIB table info                :     25 *         24\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Zebra VRF table               :     25 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Nexthop tracking object       :      2 *        240\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  ZEBRA VRF                     :      5 *       5016\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  Other Table                   :      4 *         56\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] write(2, "core_handler: memstats:  MH global info                :      1 *        128\n", 77) = -1 EBADF (Bad file descriptor)
[pid  4876] munmap(0x7f3f783d6000, 8192) = 0
[pid  4876] gettid()                    = 4876
[pid  4876] unlinkat(3, "logbuf.4876", 0) = 0
[pid  4876] rt_sigprocmask(SIG_UNBLOCK, [SEGV ALRM], NULL, 8) = 0
[pid  4876] tgkill(4876, 4876, SIGSEGV) = 0
[pid  4876] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_TKILL, si_pid=4876, si_uid=92} ---
[pid  4888] <... ppoll resumed> <unfinished ...>) = ?
[pid  4885] <... ppoll resumed> <unfinished ...>) = ?
[pid  4879] <... ppoll resumed> <unfinished ...>) = ?
[pid  4878] <... ppoll resumed> <unfinished ...>) = ?
[pid  4877] <... futex resumed>)        = ?
[pid  4888] +++ killed by SIGSEGV +++
[pid  4885] +++ killed by SIGSEGV +++
[pid  4879] +++ killed by SIGSEGV +++
[pid  4878] +++ killed by SIGSEGV +++
[pid  4877] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++

The issue was first observed with FRR 8.0 and confirmed with 9.1.
OS: CentOS 7.8
Kernel: 3.10.0-1127.19.1.el7.x86_64

I'm ready to provide additional info if needed for debug.
@pguibert6WIND
Copy link
Member

Hi, not reproduced using the inputs of this ticket on upstream.
is it possible to get the back trace from GDB ?

@odivlad
Copy link
Contributor Author

odivlad commented Jan 19, 2024 •
edited
Loading

Hi, thanks for the response!

(gdb) bt
#0  zvrf_id (zvrf=0x1bdf8a0) at zebra/zebra_vrf.h:207
#1  rib_process (rn=0x1bdd040) at zebra/zebra_rib.c:1127
#2  process_subq_route (qindex=2 '\002', lnode=0x1bdd160) at zebra/zebra_rib.c:2386
#3  process_subq (qindex=<optimized out>, subq=0x1a00760) at zebra/zebra_rib.c:2428
#4  meta_queue_process (dummy=<optimized out>, data=0x1a00e20) at zebra/zebra_rib.c:2461
#5  0x00007f9a6cad2630 in work_queue_run (thread=0x7ffcb4558aa0) at lib/workqueue.c:291
#6  0x00007f9a6cac853b in thread_call (thread=thread@entry=0x7ffcb4558aa0) at lib/thread.c:1825
#7  0x00007f9a6ca888f0 in frr_run (master=0x184d9e0) at lib/libfrr.c:1155
#8  0x000000000041e868 in main (argc=9, argv=0x7ffcb4558e88) at zebra/main.c:470

Please, note, this backtrace is collected within frr frr-8.0.1-02.el7.x86_64.rpm, 'cause my environment uses it.

@odivlad
Copy link
Contributor Author

odivlad commented Jan 24, 2024

@pguibert6WIND, hi! Do you need any additional information?

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

@frrbot
Copy link

frrbot bot commented Jul 23, 2024

This issue will be automatically closed in the specified period unless there is further activity.

@odivlad
Copy link
Contributor Author

odivlad commented Jul 23, 2024

The problem persists.

@frrbot
Copy link

frrbot bot commented Jul 23, 2024

This issue will no longer be automatically closed.

@frrbot frrbot bot removed the autoclose label Jul 23, 2024
@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

@frrbot
Copy link

frrbot bot commented Jan 20, 2025

This issue will be automatically closed in the specified period unless there is further activity.

@frrbot frrbot bot closed this as completed Jan 27, 2025
@frrbot frrbot bot removed the autoclose label Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pguibert6WIND pguibert6WIND

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@odivlad @pguibert6WIND