Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

With 256 BGP sessions between two devices, few are not established when the peer router is rebooted #17655

Open
2 tasks done
dgsudharsan opened this issue Dec 16, 2024 · 6 comments
Labels
bgp triage Needs further investigation

Comments

@dgsudharsan
Copy link

dgsudharsan commented Dec 16, 2024

Description

When there are 256 BGP sessions between two devices, when one of them is reboot and comes back up, few sessions are not established.

This issue is seen only with the below graceful restart config. Without this issue is not seen

 bgp graceful-restart restart-time 240
 bgp graceful-restart
 bgp graceful-restart preserve-fw-state
bgp graceful-restart select-defer-time 45
vtysh -c "show bgp summary"

IPv6 Unicast Summary:
BGP router identifier 10.1.0.1, local AS number 20 VRF default vrf-id 0
BGP table version 1198000
RIB entries 1999, using 250 KiB of memory
Peers 257, using 5168 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
2001:db8:1a::1  4         10        23        89  1197454    0    0 00:13:22         1000     1000 SW1
2001:db8:1b::1  4         10        22        30  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:1c::1  4         10       512       681        0    0    0 00:00:01            0        0 SW1
2001:db8:1d::1  4         10        23        89  1197454    0    0 00:13:21         1000     1000 SW1
2001:db8:1e::1  4         10        22        88  1197454    0    0 00:13:21         1000     1000 SW1
2001:db8:1f::1  4         10        22        88  1197454    0    0 00:13:18         1000     1000 SW1
2001:db8:20::1  4         10       529       684        0    0    0 00:00:00            0        0 SW1
2001:db8:21::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:22::1  4         10        23        31  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:23::1  4         10        23        31  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:24::1  4         10        23        89  1197454    0    0 00:13:22         1000     1000 SW1
2001:db8:25::1  4         10        22        30  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:26::1  4         10        23        31  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:27::1  4         10        24        32  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:28::1  4         10        23        89  1197454    0    0 00:13:22         1000     1000 SW1
2001:db8:29::1  4         10        22        88  1197454    0    0 00:13:22         1000     1000 SW1
2001:db8:2a::1  4         10       523       685        0    0    0 00:00:01            0        0 SW1
2001:db8:2b::1  4         10       524       678        0    0    0 00:00:00            0        0 SW1
2001:db8:2c::1  4         10       525       681        0    0    0 00:00:00            0        0 SW1
2001:db8:2d::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:2e::1  4         10       517       684        0    0    0 00:00:01            0        0 SW1
2001:db8:2f::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:30::1  4         10        22        88  1197454    0    0 00:13:18         1000     1000 SW1
2001:db8:31::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:32::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:33::1  4         10       524       660        0    0    0 00:00:01      Connect        0 SW1
2001:db8:34::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:35::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:36::1  4         10        25        25  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:37::1  4         10        22        88  1197454    0    0 00:13:18         1000     1000 SW1
2001:db8:38::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:39::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:3a::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:3b::1  4         10        25        25  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:3c::1  4         10       509       685        0    0    0 00:00:02            0        0 SW1
2001:db8:3d::1  4         10       244       397  1197454    0    0 00:07:07         1000     1000 SW1
2001:db8:3e::1  4         10        21        86  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:3f::1  4         10        27        27  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:40::1  4         10        21        86  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:41::1  4         10        21        86  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:42::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:43::1  4         10       506       691        0    0    0 00:00:01            0        0 SW1
2001:db8:44::1  4         10        25        25  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:45::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:46::1  4         10        25        25  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:47::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:48::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:49::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:4a::1  4         10        21        86  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:4b::1  4         10        21        86  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:4c::1  4         10        22        87  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:4d::1  4         10        24        88  1197454    0    0 00:12:52         1000     1000 SW1
2001:db8:4e::1  4         10        47        47  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:4f::1  4         10        22        30  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:50::1  4         10        23        31  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:51::1  4         10        23        31  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:52::1  4         10        23        31  1197454    0    0 00:13:50         1000     1000 SW1
2001:db8:53::1  4         10       519       599        0    0    0 00:00:01         Idle        0 SW1
2001:db8:54::1  4         10       522       615        0    0    0 00:00:01         Idle        0 SW1
2001:db8:55::1  4         10        23        31  1197454    0    0 00:13:50         1000     1000 SW1
2001:db8:56::1  4         10        47        47  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:57::1  4         10       514       678        0    0    0 00:00:01            0        0 SW1
2001:db8:58::1  4         10        23        31  1197454    0    0 00:13:50         1000     1000 SW1
2001:db8:59::1  4         10        23        31  1197454    0    0 00:13:49         1000     1000 SW1
2001:db8:5a::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:5b::1  4         10       520       673        0    0    0 00:00:01            0        0 SW1
2001:db8:5c::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:5d::1  4         10        22        88  1197454    0    0 00:13:17         1000     1000 SW1
2001:db8:5e::1  4         10       501       668        0    0    0 00:00:02            0        0 SW1
2001:db8:5f::1  4         10       513       638        0    0    0 00:00:01         Idle        0 SW1
2001:db8:60::1  4         10       505       651        0    0    0 00:00:02            0        0 SW1
2001:db8:61::1  4         10       510       655        0    0    0 00:00:00         Idle        0 SW1
2001:db8:62::1  4         10       506       671        0    0    0 00:00:02            0        0 SW1
2001:db8:63::1  4         10       511       659        0    0    0 00:00:00         Idle        0 SW1
2001:db8:64::1  4         10       506       652        0    0    0 00:00:02            0        0 SW1
2001:db8:65::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:66::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:67::1  4         10        23        89  1197454    0    0 00:13:12         1000     1000 SW1
2001:db8:68::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:69::1  4         10        22        88  1197454    0    0 00:13:12         1000     1000 SW1
2001:db8:6a::1  4         10       505       672        0    0    0 00:00:00         Idle        0 SW1
2001:db8:6b::1  4         10        22        30  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:6c::1  4         10       514       654        0    0    0 00:00:01         Idle        0 SW1
2001:db8:6d::1  4         10        23        23  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:6e::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:6f::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:70::1  4         10        24        24  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:71::1  4         10        23        23  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:72::1  4         10       507       670        0    0    0 00:00:02            0        0 SW1
2001:db8:73::1  4         10       510       668        0    0    0 00:00:01         Idle        0 SW1
2001:db8:74::1  4         10        22        22  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:75::1  4         10       508       682        0    0    0 00:00:02            0        0 SW1
2001:db8:76::1  4         10       504       659        0    0    0 00:00:02            0        0 SW1
2001:db8:77::1  4         10       515       645        0    0    0 00:00:00         Idle        0 SW1
2001:db8:78::1  4         10        22        22  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:79::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:7a::1  4         10       519       626        0    0    0 00:00:01         Idle        0 SW1
2001:db8:7b::1  4         10       517       694        0    0    0 00:00:01            0        0 SW1
2001:db8:7c::1  4         10        22        88  1197454    0    0 00:13:17         1000     1000 SW1
2001:db8:7d::1  4         10        23        31  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:7e::1  4         10        23        31  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:7f::1  4         10        22        90  1197454    0    0 00:13:11         1000     1000 SW1
2001:db8:80::1  4         10        22        90  1197454    0    0 00:13:11         1000     1000 SW1
2001:db8:81::1  4         10       505       675        0    0    0 00:00:00         Idle        0 SW1
2001:db8:82::1  4         10        22        22  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:83::1  4         10        47        47  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:84::1  4         10        23        23  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:85::1  4         10       503       690        0    0    0 00:00:00         Idle        0 SW1
2001:db8:86::1  4         10        25        25  1197454    0    0 00:13:40         1000     1000 SW1
2001:db8:87::1  4         10       503       694        0    0    0 00:00:00         Idle        0 SW1
2001:db8:88::1  4         10       501       708        0    0    0 00:00:00         Idle        0 SW1
2001:db8:89::1  4         10       504       655        0    0    0 00:00:00         Idle        0 SW1
2001:db8:8a::1  4         10        22        30  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:8b::1  4         10        23        31  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:8c::1  4         10        22        30  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:8d::1  4         10        23        31  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:8e::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:8f::1  4         10        23        89  1197454    0    0 00:13:17         1000     1000 SW1
2001:db8:90::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:91::1  4         10        46        46  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:92::1  4         10        24        89  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:93::1  4         10        22        88  1197454    0    0 00:13:15         1000     1000 SW1
2001:db8:94::1  4         10        24        89  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:95::1  4         10        22        87  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:96::1  4         10        22        87  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:97::1  4         10        49        50  1197454    0    0 00:13:33         1000     1000 SW1
2001:db8:98::1  4         10        21        86  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:99::1  4         10        22        90  1197454    0    0 00:13:11         1000     1000 SW1
2001:db8:9a::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:9b::1  4         10        46        47  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:9c::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:9d::1  4         10       527       687        0    0    0 00:00:00            0        0 SW1
2001:db8:9e::1  4         10       515       682        0    0    0 00:00:00         Idle        0 SW1
2001:db8:9f::1  4         10        22        30  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:a0::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:a1::1  4         10        23        31  1197454    0    0 00:13:50         1000     1000 SW1
2001:db8:a2::1  4         10        25        25  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:a3::1  4         10        22        90  1197454    0    0 00:13:08         1000     1000 SW1
2001:db8:a4::1  4         10       515       673        0    0    0 00:00:00         Idle        0 SW1
2001:db8:a5::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:a6::1  4         10       507       666        0    0    0 00:00:02            0        0 SW1
2001:db8:a7::1  4         10       510       653        0    0    0 00:00:00         Idle        0 SW1
2001:db8:a8::1  4         10        22        90  1197454    0    0 00:13:05         1000     1000 SW1
2001:db8:a9::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:aa::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:ab::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:ac::1  4         10       512       669        0    0    0 00:00:00         Idle        0 SW1
2001:db8:ad::1  4         10        23        91  1197454    0    0 00:13:12         1000     1000 SW1
2001:db8:ae::1  4         10       511       682        0    0    0 00:00:01            0        0 SW1
2001:db8:af::1  4         10        22        88  1197454    0    0 00:13:16         1000     1000 SW1
2001:db8:b0::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:b1::1  4         10        23        31  1197454    0    0 00:13:50         1000     1000 SW1
2001:db8:b2::1  4         10        22        22  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:b3::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:b4::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:b5::1  4         10        22        90  1197454    0    0 00:13:05         1000     1000 SW1
2001:db8:b6::1  4         10        22        90  1197454    0    0 00:13:05         1000     1000 SW1
2001:db8:b7::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:b8::1  4         10        23        91  1197454    0    0 00:13:05         1000     1000 SW1
2001:db8:b9::1  4         10        23        23  1197454    0    0 00:13:36         1000     1000 SW1
2001:db8:ba::1  4         10        23        88  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:bb::1  4         10        22        87  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:bc::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:bd::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:be::1  4         10        22        87  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:bf::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c0::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c1::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c2::1  4         10        21        86  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:c3::1  4         10        24        24  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c4::1  4         10        25        24  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c5::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c6::1  4         10       506       667        0    0    0 00:00:02            0        0 SW1
2001:db8:c7::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c8::1  4         10        24        24  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:c9::1  4         10        24        89  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:ca::1  4         10        25        93  1197454    0    0 00:13:11         1000     1000 SW1
2001:db8:cb::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:cc::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:cd::1  4         10        25        93  1197454    0    0 00:13:11         1000     1000 SW1
2001:db8:ce::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:cf::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d0::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d1::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d2::1  4         10        22        87  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:d3::1  4         10        23        88  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:d4::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d5::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d6::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d7::1  4         10        25        90  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:d8::1  4         10        23        23  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:d9::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:da::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:db::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:dc::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:dd::1  4         10        25        90  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:de::1  4         10        25        90  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:df::1  4         10        22        87  1197454    0    0 00:13:00         1000     1000 SW1
2001:db8:e0::1  4         10        25        90  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:e1::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:e2::1  4         10        21        86  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:e3::1  4         10        24        89  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:e4::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:e5::1  4         10        22        87  1197454    0    0 00:12:53         1000     1000 SW1
2001:db8:e6::1  4         10        24        24  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:e7::1  4         10        21        86  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:e8::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:e9::1  4         10        21        86  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:ea::1  4         10        22        22  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:eb::1  4         10        22        87  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:ec::1  4         10        25        90  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:ed::1  4         10        25        89  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:ee::1  4         10        25        25  1197454    0    0 00:13:35         1000     1000 SW1
2001:db8:ef::1  4         10        23        87  1197454    0    0 00:11:41         1000     1000 SW1
2001:db8:f0::1  4         10        24        89  1197454    0    0 00:12:57         1000     1000 SW1
2001:db8:f1::1  4         10        21        86  1197454    0    0 00:12:56         1000     1000 SW1
2001:db8:f2::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:f3::1  4         10        22        87  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:f4::1  4         10        23        88  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:f5::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:f6::1  4         10        22        22  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:f7::1  4         10        22        87  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:f8::1  4         10        21        86  1197454    0    0 00:12:58         1000     1000 SW1
2001:db8:f9::1  4         10        23        88  1197454    0    0 00:12:59         1000     1000 SW1
2001:db8:fa::1  4         10        23        31  1197454    0    0 00:13:49         1000     1000 SW1
2001:db8:fb::1  4         10       527       684        0    0    0 00:00:00            0        0 SW1
2001:db8:fc::1  4         10        23        31  1197454    0    0 00:13:49         1000     1000 SW1
2001:db8:fd::1  4         10       518       686        0    0    0 00:00:01            0        0 SW1
2001:db8:fe::1  4         10       520       650        0    0    0 00:00:01         Idle        0 SW1
2001:db8:ff::1  4         10       529       691        0    0    0 00:00:00            0        0 SW1
2001:db8:100::1 4         10        23        89  1197454    0    0 00:13:18         1000     1000 SW1
2001:db8:101::1 4         10        22        30  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:102::1 4         10        22        88  1197454    0    0 00:13:17         1000     1000 SW1
2001:db8:103::1 4         10        23        31  1197454    0    0 00:13:49         1000     1000 SW1
2001:db8:104::1 4         10       519       644        0    0    0 00:00:01         Idle        0 SW1
2001:db8:105::1 4         10        22        88  1197454    0    0 00:13:17         1000     1000 SW1
2001:db8:106::1 4         10       463       639        0    0    0 00:00:01            0        0 SW1
2001:db8:107::1 4         10        22        88  1197454    0    0 00:13:17         1000     1000 SW1
2001:db8:108::1 4         10        23        31  1197454    0    0 00:13:50         1000     1000 SW1
2001:db8:109::1 4         10       525       627        0    0    0 00:00:00            0        0 SW1
2001:db8:10a::1 4         10        23        31  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:10b::1 4         10        23        31  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:10c::1 4         10        23        23  1197454    0    0 00:13:34         1000     1000 SW1
2001:db8:10d::1 4         10        23        89  1197454    0    0 00:13:21         1000     1000 SW1
2001:db8:10e::1 4         10        22        30  1197454    0    0 00:13:52         1000     1000 SW1
2001:db8:10f::1 4         10        22        30  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:110::1 4         10        22        30  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:111::1 4         10       517       703        0    0    0 00:00:01            0        0 SW1
2001:db8:112::1 4         10        22        88  1197454    0    0 00:13:22         1000     1000 SW1
2001:db8:113::1 4         10        22        30  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:114::1 4         10        22        30  1197454    0    0 00:13:51         1000     1000 SW1
2001:db8:115::1 4         10        23        31  1197454    0    0 00:13:45         1000     1000 SW1
2001:db8:116::1 4         10        22        88  1197454    0    0 00:13:21         1000     1000 SW1
2001:db8:117::1 4         10        22        30  1197454    0    0 00:13:49         1000     1000 SW1
2001:db8:118::1 4         10        22        88  1197454    0    0 00:13:20         1000     1000 SW1
2001:db8:119::1 4         10        23        31  1197454    0    0 00:13:49         1000     1000 SW1

In addition issue results in 100% utilization for bgpd

top - 21:53:14 up 26 min,  1 user,  load average: 3.74, 4.54, 3.64
Threads: 829 total,   4 running, 819 sleeping,   0 stopped,   6 zombie
%Cpu(s): 11.4 us,  4.8 sy,  0.0 ni, 83.6 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  31913.1 total,  20775.8 free,   5486.4 used,   6370.4 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  26426.7 avail Mem
expand which numa node (0-0)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  11858 300       20   0  501540 268216  14368 R  99.9   0.8  31:47.82 bgpd
  11867 300       20   0  501540 268216  14368 R  51.7   0.8  10:51.40 bgpd_io

Attaching the running config.
show_run.txt

Version

FRR 10.0.1 with SONiC 202411.

How to reproduce

Create 256 sessions between two devices and reboot one of the device after sessions are established.

Expected behavior

All sessions should be established when peer device reboot

Actual behavior

Few of the 256 sessions are not established.

Additional context

Donald did some investigation and found the below

union sockunion *sockunion_getpeername(int fd) { int ret; socklen_t len; union { struct sockaddr sa; struct sockaddr_in sin; struct sockaddr_in6 sin6; char tmp_buffer[128]; } name; union sockunion *su; memset(&name, 0, sizeof(name)); len = sizeof(name); ret = getpeername(fd, (struct sockaddr *)&name, &len); if (ret < 0) { flog_err(EC_LIB_SOCKET, "Can't get remote address and port: %s", safe_strerror(errno)); return NULL; }

so return is 0 and we are getting name set to this:

(gdb) p name $14 = {sa = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = { sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, tmp_buffer = '\000' <repeats 127 times>} (gdb) p len $15 = 128

Checklist

  • I have searched the open issues for this bug.
  • I have not included sensitive information in this report.
@dgsudharsan dgsudharsan added the triage Needs further investigation label Dec 16, 2024
@dgsudharsan
Copy link
Author

@donaldsharp FYI

@dgsudharsan
Copy link
Author

Here is the show bgp neighbor output in problem state on both devices for few neighbors

Device 1 (Device that is not reloaded and has bgpd 100%)

vtysh -c "show bgp neighbor 2001:db8:20::1"
BGP neighbor is 2001:db8:20::1, remote AS 10, local AS 20, external link
  Local Role: undefined
  Remote Role: undefined
 Description: SW1
Hostname: sonic
 Member of peer-group PEER_V6 for session parameters
  BGP version 4, remote router ID 10.1.0.1, local router ID 10.1.0.1
  BGP state = Idle
  Last read 00:00:01, Last write 00:00:01
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    The remaining time of restart timer is 239
    The remaining time of stalepath timer is 359
    Local GR Mode: Restart*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 240
      Received Restart Time(sec): 240
      Configured LLGR Stale Path Time(sec): 0
      Restart Time Remaining(sec): 239
      Restart Time Remaining(sec): 239
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:               3566       3566
    Notifications:          0          0
    Updates:              166         42
    Keepalives:          3568       3568
    Route Refresh:          0          0
    Capability:             0          0
    Total:               7300       7176
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv6 Unicast
  PEER_V6 peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Local AS allowed in path, 1 occurrences
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *FROM_BGP_PEER_V6
  Route map for outgoing advertisements is *TO_BGP_PEER_V6
  0 accepted prefixes

  Connections established 3566; dropped 3566
  Last reset 00:00:00,   NSF peer closed the session (n/a)
  External BGP neighbor may be up to 1 hops away.
Local host: 2001:db8:20::2, Local port: 40756
Foreign host: 2001:db8:20::1, Foreign port: 179
Nexthop: 10.1.0.1
Nexthop global: 2001:db8:20::2
Nexthop local: fe80::b2cf:eff:fe6d:2c00
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next start timer due in 0 seconds
Read thread: off  Write thread: off  FD used: -1

Device 2 for the neighbor above

vtysh -c " show bgp neighbor 2001:db8:20::2"
BGP neighbor is 2001:db8:20::2, remote AS 20, local AS 10, external link
  Local Role: undefined
  Remote Role: undefined
 Description: SW2
Hostname: sonic
 Member of peer-group PEER_V6 for session parameters
  BGP version 4, remote router ID 10.1.0.1, local router ID 10.1.0.1
  BGP state = Idle
  Last read 01:49:38, Last write 00:00:01
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    The remaining time of restart timer is 239
    The remaining time of stalepath timer is 359
    Local GR Mode: Restart*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 240
      Received Restart Time(sec): 240
      Configured LLGR Stale Path Time(sec): 0
      Restart Time Remaining(sec): 239
      Restart Time Remaining(sec): 239
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:               7105       3552
    Notifications:       3552          2
    Updates:               42        164
    Keepalives:          3552       3552
    Route Refresh:          0          0
    Capability:             0          0
    Total:              14251       7270
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv6 Unicast
  PEER_V6 peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Local AS allowed in path, 1 occurrences
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *FROM_BGP_PEER_V6
  Route map for outgoing advertisements is *TO_BGP_PEER_V6
  0 accepted prefixes

  Connections established 3552; dropped 3552
  Last reset 00:00:01,   NSF peer closed the session (n/a)
  External BGP neighbor may be up to 1 hops away.
Local host: 2001:db8:20::1, Local port: 179
Foreign host: 2001:db8:20::2, Foreign port: 34790
Nexthop: 10.1.0.1
Nexthop global: 2001:db8:20::1
Nexthop local: fe80::9e05:91ff:fea0:ca00
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next start timer due in 0 seconds
Read thread: off  Write thread: off  FD used: -1

Device 1 second neighbor:

vtysh -c "show bgp neighbor 2001:db8:2f::1"
BGP neighbor is 2001:db8:2f::1, remote AS 10, local AS 20, external link
  Local Role: undefined
  Remote Role: undefined
 Description: SW1
Hostname: sonic
 Member of peer-group PEER_V6 for session parameters
  BGP version 4, remote router ID 10.1.0.1, local router ID 10.1.0.1
  BGP state = Idle
  Last read 00:00:01, Last write 00:00:01
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    The remaining time of restart timer is 239
    The remaining time of stalepath timer is 359
    Local GR Mode: Restart*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 240
      Received Restart Time(sec): 240
      Configured LLGR Stale Path Time(sec): 0
      Restart Time Remaining(sec): 239
      Restart Time Remaining(sec): 239
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:               3637       3636
    Notifications:          1          2
    Updates:              128         46
    Keepalives:          3637       3637
    Route Refresh:          0          0
    Capability:             0          0
    Total:               7403       7321
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv6 Unicast
  PEER_V6 peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Local AS allowed in path, 1 occurrences
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *FROM_BGP_PEER_V6
  Route map for outgoing advertisements is *TO_BGP_PEER_V6
  0 accepted prefixes

  Connections established 3636; dropped 3636
  Last reset 00:00:00,   NSF peer closed the session (n/a)
  External BGP neighbor may be up to 1 hops away.
Local host: 2001:db8:2f::2, Local port: 39680
Foreign host: 2001:db8:2f::1, Foreign port: 179
Nexthop: 10.1.0.1
Nexthop global: 2001:db8:2f::2
Nexthop local: fe80::b2cf:eff:fe6d:2c00
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next start timer due in 0 seconds
Read thread: off  Write thread: off  FD used: -1

Device 2 2nd neighbor in problem state

vtysh -c "show bgp neighbor 2001:db8:2f::2"
BGP neighbor is 2001:db8:2f::2, remote AS 20, local AS 10, external link
  Local Role: undefined
  Remote Role: undefined
 Description: SW2
Hostname: sonic
 Member of peer-group PEER_V6 for session parameters
  BGP version 4, remote router ID 10.1.0.1, local router ID 10.1.0.1
  BGP state = Idle
  Last read 01:52:19, Last write 00:00:02
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    The remaining time of restart timer is 239
    The remaining time of stalepath timer is 359
    Local GR Mode: Restart*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 240
      Received Restart Time(sec): 240
      Configured LLGR Stale Path Time(sec): 0
      Restart Time Remaining(sec): 239
      Restart Time Remaining(sec): 239
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:               7280       3639
    Notifications:       3640          4
    Updates:               44        126
    Keepalives:          3639       3639
    Route Refresh:          0          0
    Capability:             0          0
    Total:              14603       7408
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv6 Unicast
  PEER_V6 peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Local AS allowed in path, 1 occurrences
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Outbound path policy configured
  Route map for incoming advertisements is *FROM_BGP_PEER_V6
  Route map for outgoing advertisements is *TO_BGP_PEER_V6
  0 accepted prefixes

  Connections established 3639; dropped 3639
  Last reset 00:00:01,   NSF peer closed the session (n/a)
  External BGP neighbor may be up to 1 hops away.
Local host: 2001:db8:2f::1, Local port: 179
Foreign host: 2001:db8:2f::2, Foreign port: 44164
Nexthop: 10.1.0.1
Nexthop global: 2001:db8:2f::1
Nexthop local: fe80::9e05:91ff:fea0:ca00
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next start timer due in 0 seconds
Read thread: off  Write thread: off  FD used: -1

@ton31337
Copy link
Member

Can we get the debug logging?

@ton31337 ton31337 added the bgp label Dec 17, 2024
@dgsudharsan
Copy link
Author

bgp.summary.t1.log
bgp.summary.t2.log
t1_bgpd.log
t2_bgpd.log
Attaching bgp summary output from t1 and t2. T1 device is rebooted in the scenario. We can see around 40 sessions are not established

2001:db8:1b::1  4         10      1845      2011        0    0    0 00:00:00            0        0 SW1
2001:db8:1d::1  4         10      2927      3067        0    0    0 00:00:00            0        0 SW1
2001:db8:1e::1  4         10      2006      2169        0    0    0 00:00:01      Connect        0 SW1
2001:db8:20::1  4         10      2929      3064        0    0    0 00:00:01         Idle        0 SW1
2001:db8:22::1  4         10      1845      2025        0    0    0 00:00:00            0        0 SW1
2001:db8:23::1  4         10      1843      1992        0    0    0 00:00:00            0        0 SW1
2001:db8:26::1  4         10      1845      2008        0    0    0 00:00:00            0        0 SW1
2001:db8:28::1  4         10      3202      3330        0    0    0 00:00:00            0        0 SW1
2001:db8:2b::1  4         10      1845      2011        0    0    0 00:00:01         Idle        0 SW1
2001:db8:2e::1  4         10      2925      3066        0    0    0 00:00:01         Idle        0 SW1
2001:db8:39::1  4         10      2919      3054        0    0    0 00:00:01         Idle        0 SW1
2001:db8:4d::1  4         10      1828      2019        0    0    0 00:00:01            0        0 SW1
2001:db8:8e::1  4         10      1830      2028        0    0    0 00:00:01            0        0 SW1
2001:db8:90::1  4         10      1838      2036        0    0    0 00:00:01         Idle        0 SW1
2001:db8:92::1  4         10      1829      1995        0    0    0 00:00:01         Idle        0 SW1
2001:db8:95::1  4         10      1839      2005        0    0    0 00:00:01         Idle        0 SW1
2001:db8:97::1  4         10      2921      3048        0    0    0 00:00:00         Idle        0 SW1
2001:db8:99::1  4         10      2922      3066        0    0    0 00:00:00         Idle        0 SW1
2001:db8:ac::1  4         10      1837      2000        0    0    0 00:00:01         Idle        0 SW1
2001:db8:af::1  4         10      2103      2245        0    0    0 00:00:01         Idle        0 SW1
2001:db8:b1::1  4         10      2921      3057        0    0    0 00:00:01         Idle        0 SW1
2001:db8:be::1  4         10      1823      1989        0    0    0 00:00:01            0        0 SW1
2001:db8:ca::1  4         10      2089      2241        0    0    0 00:00:01         Idle        0 SW1
2001:db8:cc::1  4         10      2092      2229        0    0    0 00:00:00         Idle        0 SW1
2001:db8:cf::1  4         10      2087      2250        0    0    0 00:00:01         Idle        0 SW1
2001:db8:d0::1  4         10      1833      2032        0    0    0 00:00:00         Idle        0 SW1
2001:db8:d1::1  4         10      2090      2276        0    0    0 00:00:00         Idle        0 SW1
2001:db8:ea::1  4         10      1832      1999        0    0    0 00:00:00         Idle        0 SW1
2001:db8:ec::1  4         10      2093      2260        0    0    0 00:00:01            0        0 SW1
2001:db8:f1::1  4         10      1895      2071        0    0    0 00:00:01            0        0 SW1
2001:db8:f4::1  4         10      1830      2030        0    0    0 00:00:00         Idle        0 SW1
2001:db8:fa::1  4         10      2112      2243        0    0    0 00:00:01         Idle        0 SW1
2001:db8:101::1 4         10      2926      3055        0    0    0 00:00:01         Idle        0 SW1
2001:db8:10a::1 4         10      1845      2027        0    0    0 00:00:00            0        0 SW1
2001:db8:10c::1 4         10      1903      2060        0    2    0 00:00:01     OpenSent        0 SW1
2001:db8:10d::1 4         10      3086      3229        0    0    0 00:00:01         Idle        0 SW1
2001:db8:10f::1 4         10      3190      3279        0    0    0 00:00:01         Idle        0 SW1
2001:db8:110::1 4         10      1844      1994        0    0    0 00:00:01      Connect        0 SW1
2001:db8:113::1 4         10      1845      2011        0    0    0 00:00:01         Idle        0 SW1
2001:db8:119::1 4         10      2108      2231        0    0    0 00:00:01         Idle        0 SW1

@ton31337
Copy link
Member

Could you give some specific time when an actual restart happened? And as I understand this is happening ONLY if bgp graceful-restart select-defer-time is configured? Btw, could you also enable GR logging debug bgp graceful-restart?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bgp triage Needs further investigation
Projects
None yet
Development

No branches or pull requests

3 participants
@ton31337 @dgsudharsan and others