BGP with BFD cannot recover from AWS BGP bring down test: incoming conn rejected - no AF activated for peer

[kyleli666]

Background

FRR Version 10.2.1-0~ubuntu24.04.1
Linux 6.8.1-1014-realtime #15-Ubuntu SMP PREEMPT_RT Fri Dec 6 22:24:34 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

I created a BGP with BFD to AWS Direct Connection and would like to test the failover to AWS VPN by a test function from AWS "Bring down BGP".
Both BGP and BFD establish well.
However, when testing, the failover works well, but when recovering, BGP keeps Idle and BFD keeps down.

Jan 17 14:21:15 node1 bfdd[3055688]: [SEY1D-NT8EQ] state-change: [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622] up -> down reason:neighbor-down
Jan 17 14:21:15 node1 bgpd[3055678]: [Q4BCV-6FHZ5] zclient_bfd_session_update: 169.254.66.2/32 -> 169.254.66.1/32 (interface vlan1622) VRF default(0) (CPI bit yes): Down
Jan 17 14:21:15 node1 bgpd[3055678]: [MKVHZ-7MS3V] bfd_session_status_update: neighbor 169.254.66.1 vrf default(0) bfd state Up -> Down
Jan 17 14:21:15 node1 bgpd[3055678]: [HZN6M-XRM1G] %NOTIFICATION(Hard Reset): sent to neighbor 169.254.66.1 6/10 (Cease/BFD Down) 0 bytes
Jan 17 14:21:15 node1 bgpd[3055678]: [QFMSE-NPSNN] zclient_bfd_session_update:   sessions updated: 1
Jan 17 14:21:15 node1 bgpd[3055678]: [ZWCSR-M7FG9] 169.254.66.1 [FSM] BGP_Stop (Established->Clearing), fd 25
Jan 17 14:21:15 node1 bgpd[3055678]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Clearing established_peers 2
Jan 17 14:21:15 node1 bgpd[3055678]: [HKWM3-ZC5QP] 169.254.66.1 fd -1 went from Established to Clearing
Jan 17 14:21:15 node1 bgpd[3055678]: [ZWCSR-M7FG9] 169.254.66.1 [FSM] Clearing_Completed (Clearing->Idle), fd -1
Jan 17 14:21:15 node1 bgpd[3055678]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Idle established_peers 2
Jan 17 14:21:15 node1 bgpd[3055678]: [HKWM3-ZC5QP] 169.254.66.1 fd -1 went from Clearing to Idle
Jan 17 14:22:08 node1 bgpd[3055678]: [ZGYKZ-X9JJR] 169.254.66.1 - incoming conn rejected - no AF activated for peer
Jan 17 14:22:31 node1 bgpd[3055678]: [ZGYKZ-X9JJR] 169.254.66.1 - incoming conn rejected - no AF activated for peer
Jan 17 14:22:33 node1 bgpd[3055678]: [ZGYKZ-X9JJR] 169.254.66.1 - incoming conn rejected - no AF activated for peer

Current configuration:
!
frr version 10.2.1
frr defaults traditional
hostname node1
log syslog
service integrated-vtysh-config
!
debug bgp bfd
debug bfd peer
debug bfd zebra
!
ip router-id xxx.xx.xxx.xx
!
router bgp 65031
 no bgp ebgp-requires-policy
 no bgp hard-administrative-reset
 neighbor 169.254.44.1 remote-as 64512
 neighbor 169.254.44.1 description BGP VPN tunnel1
 neighbor 169.254.44.1 update-source 169.254.44.2
 neighbor 169.254.55.1 remote-as 64512
 neighbor 169.254.55.1 description BGP VPN tunnel2
 neighbor 169.254.55.1 update-source 169.254.55.2
 neighbor 169.254.66.1 remote-as 65030
 neighbor 169.254.66.1 bfd
 neighbor 169.254.66.1 description BGP Line
 neighbor 169.254.66.1 update-source 169.254.66.2
 !
 address-family ipv4 unicast
  network 169.254.44.0/30
  network 169.254.55.0/30
  network 169.254.66.0/30
  neighbor 169.254.44.1 weight 100
  neighbor 169.254.55.1 weight 200
  neighbor 169.254.66.1 weight 300
 exit-address-family
exit
!
end

Current Workaround

If I do not enable BFD, BGP can recover well, but I believe it is slower when failover in real cases.

Try 1

AI suggests me to config "neighbor 169.254.66.1 activate" under "address-family ipv4 unicast", but it seems default, not show up in "show run" and not solving the issue, and still "no AF activated for peer"

Try 2

I had a very similar setup with VyOS which uses FRR 9.1.1, and that recovers well.
I looked into the frr.conf of this VyOS, and manually update my frr.conf accordingly, and finally neighbor activate is there by vim...

router bgp 65031
 no bgp ebgp-requires-policy
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor 169.254.44.1 remote-as 64512
 neighbor 169.254.44.1 description BGP VPN tunnel1
 neighbor 169.254.44.1 update-source 169.254.44.2
 neighbor 169.254.55.1 remote-as 64512
 neighbor 169.254.55.1 description BGP VPN tunnel2
 neighbor 169.254.55.1 update-source 169.254.55.2
 neighbor 169.254.66.1 remote-as 65030
 neighbor 169.254.66.1 bfd
 neighbor 169.254.66.1 description BGP Line
 neighbor 169.254.66.1 update-source 169.254.66.2
 !
 address-family ipv4 unicast
  network 169.254.44.0/30
  network 169.254.55.0/30
  network 169.254.66.0/30
  neighbor 169.254.44.1 activate
  neighbor 169.254.44.1 weight 100
  neighbor 169.254.55.1 activate
  neighbor 169.254.55.1 weight 200
  neighbor 169.254.66.1 activate
  neighbor 169.254.66.1 weight 300
 exit-address-family
exit

Now, debug log does not show "no AF activated for peer" anymore, but still cannot recover BGP until I remove BFD configuration...

Jan 18 01:27:28 node1 bgpd[3466715]: [MKVHZ-7MS3V] bfd_session_status_update: neighbor 169.254.66.1 vrf default(0) bfd state Down -> Up
Jan 18 01:27:28 node1 bgpd[3466715]: [QFMSE-NPSNN] zclient_bfd_session_update:   sessions updated: 1
Jan 18 01:27:39 node1 watchfrr[3466696]: [WFP93-1D146] configuration write completed with exit code 0
Jan 18 01:30:09 node1 bfdd[3466725]: [SEY1D-NT8EQ] state-change: [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622] up -> down reason:neighbor-down
Jan 18 01:30:09 node1 bgpd[3466715]: [Q4BCV-6FHZ5] zclient_bfd_session_update: 169.254.66.2/32 -> 169.254.66.1/32 (interface vlan1622) VRF default(0) (CPI bit yes): Down
Jan 18 01:30:09 node1 bgpd[3466715]: [MKVHZ-7MS3V] bfd_session_status_update: neighbor 169.254.66.1 vrf default(0) bfd state Up -> Down
Jan 18 01:30:09 node1 bgpd[3466715]: [QFMSE-NPSNN] zclient_bfd_session_update:   sessions updated: 1
Jan 18 01:34:08 node1 bfdd[3466725]: [MSVDW-Y8Z5Q] ptm-del-dest: deregister peer [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default cbit:0x00 minimum-ttl:255]
Jan 18 01:34:08 node1 bfdd[3466725]: [NYF5K-SE3NS] ptm-del-session: [mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622] refcount=0
Jan 18 01:34:08 node1 bfdd[3466725]: [NW21R-MRYNT] session-delete: mhop:no peer:169.254.66.1 local:169.254.66.2 vrf:default ifname:vlan1622
Jan 18 01:34:17 node1 watchfrr[3466696]: [WFP93-1D146] configuration write completed with exit code 0
Jan 18 01:34:43 node1 bgpd[3466715]: [JG0WZ-7X009][EC 33554504] 169.254.66.1 unrecognized capability code: 128 - ignored
Jan 18 01:34:44 node1 bgpd[3466715]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 169.254.66.1 in vrf default

Try 3

I finally tried graceful-restart-disble, in case of confliction, but still cannot recover BGP peering, it keeps in Idle.

Question

What could be the possible cause?