static: blackhole route in vrf will vanish on every second reload

[c-po]

Description

Using frr-reload.py on every second reload of a VRF blackhole route with distance and tag applied it will vanish from vtysh.

Version

stable/10.2 6cd056b

How to reproduce

Use

!
vrf red
 ip route 10.0.0.0/8 blackhole 250
exit-vrf
!

First reload the configuration which will add the route into the VRF block:

$ sudo /usr/lib/frr/frr-reload.py --debug --stdout --reload vrf.txt
2024-12-07 20:04:17,161  INFO: Called via "Namespace(input=None, reload=True, test=False, debug=True, log_level='info', stdout=True, pathspace=None, filename='vrf.txt', overwrite=False, bindir='/usr/bin', confdir='/etc/frr', rundir='/var/run/frr', vty_socket=None, daemon='', test_reset=False)"
2024-12-07 20:04:17,161  INFO: Loading Config object from file vrf.txt
2024-12-07 20:04:17,167 DEBUG: LINE vrf red                                           : enter context ['vrf red']
2024-12-07 20:04:17,167 DEBUG: LINE ip route 10.0.0.0/8 blackhole 250 tag 210         : add to current context ['vrf red']                                   
2024-12-07 20:04:17,168 DEBUG: LINE exit-vrf                                          : exit context ['vrf red']
2024-12-07 20:04:17,176 DEBUG: New Frr Config
!
vrf red
ip route 10.0.0.0/8 blackhole 250 tag 210
exit-vrf
end
!

end

2024-12-07 20:04:17,176  INFO: Loading Config object from vtysh show running
2024-12-07 20:04:17,184 DEBUG: LINE frr version 10.2                                  : single-line context
2024-12-07 20:04:17,185 DEBUG: LINE frr defaults traditional                          : single-line context
2024-12-07 20:04:17,185 DEBUG: LINE hostname LR1.wue3                                 : single-line context
2024-12-07 20:04:17,185 DEBUG: LINE service integrated-vtysh-config                   : single-line context
2024-12-07 20:04:17,185 DEBUG: LINE vrf red                                           : enter context ['vrf red']
2024-12-07 20:04:17,185 DEBUG: LINE exit-vrf                                          : exit context ['vrf red']
2024-12-07 20:04:17,185 DEBUG: Running Frr Config (Pass #0)
frr version 10.2
frr defaults traditional
hostname LR1.wue3
service integrated-vtysh-config
!
vrf red
exit-vrf
end
!
end
2024-12-07 20:04:17,185  INFO: "frr version 10.2" cannot be removed
2024-12-07 20:04:17,185  INFO: "frr defaults traditional" cannot be removed
2024-12-07 20:04:17,185  INFO: "service integrated-vtysh-config" cannot be removed
2024-12-07 20:04:17,192  INFO: Executed "no hostname LR1.wue3"
2024-12-07 20:04:17,193  INFO: /var/run/frr/reload-QKC7G1.txt content
['vrf red\n ip route 10.0.0.0/8 blackhole 250 tag 210\nexit\n']
[5398|mgmtd] sending configuration
[5399|zebra] sending configuration
[5400|ripd] sending configuration
[5400|ripd] done
[5401|ripngd] sending configuration
[5402|ospfd] sending configuration
[5401|ripngd] done
[5403|ospf6d] sending configuration
[5405|bgpd] sending configuration
[5404|ldpd] sending configuration
[5399|zebra] done
[5402|ospfd] done
[5398|mgmtd] done
[5406|isisd] sending configuration
[5403|ospf6d] done
[5404|ldpd] done
[5405|bgpd] done
[5410|babeld] sending configuration
[5412|fabricd] sending configuration
[5413|watchfrr] sending configuration
[5406|isisd] done
Waiting for children to finish applying config...
[5416|bfdd] sending configuration
[5415|staticd] sending configuration
[5419|pim6d] sending configuration
[5410|babeld] done
[5419|pim6d] done
[5412|fabricd] done
[5413|watchfrr] done
[5416|bfdd] done
[5415|staticd] done
2024-12-07 20:04:17,207  INFO: Loading Config object from vtysh show running
2024-12-07 20:04:17,216 DEBUG: LINE frr version 10.2                                  : single-line context
2024-12-07 20:04:17,216 DEBUG: LINE frr defaults traditional                          : single-line context
2024-12-07 20:04:17,216 DEBUG: LINE hostname LR1.wue3                                 : single-line context
2024-12-07 20:04:17,216 DEBUG: LINE service integrated-vtysh-config                   : single-line context
2024-12-07 20:04:17,216 DEBUG: LINE vrf red                                           : enter context ['vrf red']
2024-12-07 20:04:17,216 DEBUG: LINE ip route 10.0.0.0/8 blackhole tag 210 250         : add to current context ['vrf red']                                   
2024-12-07 20:04:17,216 DEBUG: LINE exit-vrf                                          : exit context ['vrf red']
2024-12-07 20:04:17,216 DEBUG: Running Frr Config (Pass #1)
frr version 10.2
frr defaults traditional
hostname LR1.wue3
service integrated-vtysh-config
!
vrf red
ip route 10.0.0.0/8 blackhole tag 210 250
exit-vrf
end
!
end
2024-12-07 20:04:17,216  INFO: "frr version 10.2" cannot be removed
2024-12-07 20:04:17,216  INFO: "frr defaults traditional" cannot be removed
2024-12-07 20:04:17,216  INFO: "service integrated-vtysh-config" cannot be removed
2024-12-07 20:04:17,217  INFO: /var/run/frr/reload-5GQL3H.txt content
['vrf red\n ip route 10.0.0.0/8 blackhole 250 tag 210\nexit\n',
 'vrf red\n no ip route 10.0.0.0/8 blackhole tag 210 250\nexit\n',
 'vrf red\n ip route 10.0.0.0/8 blackhole 250 tag 210\nexit\n']
[5423|mgmtd] sending configuration
[5425|ripd] sending configuration
[5426|ripngd] sending configuration
[5427|ospfd] sending configuration
[5428|ospf6d] sending configuration
[5424|zebra] sending configuration
[5435|babeld] sending configuration
[5431|isisd] sending configuration
[5429|ldpd] sending configuration
[5437|fabricd] sending configuration
[5430|bgpd] sending configuration
[5438|watchfrr] sending configuration
[5435|babeld] done
[5424|zebra] done
[5438|watchfrr] done
[5425|ripd] done
[5437|fabricd] done
[5440|staticd] sending configuration
Waiting for children to finish applying config...
[5441|bfdd] sending configuration
[5429|ldpd] done
[5444|pim6d] sending configuration
[5426|ripngd] done
[5440|staticd] done
[5441|bfdd] done
[5427|ospfd] done
[5431|isisd] done
MGMTD: No changes found to be committed!
[5423|mgmtd] done
[5430|bgpd] done
[5444|pim6d] done
[5428|ospf6d] done

Now reload the same file again a second time - the route will be removed

$sudo /usr/lib/frr/frr-reload.py --debug --stdout  --reload vrf.txt
2024-12-07 20:05:26,072  INFO: Called via "Namespace(input=None, reload=True, test=False, debug=True, log_level='info', stdout=True, pathspace=None, filename='vrf.txt', overwrite=False, bindir='/usr/bin', confdir='/etc/frr', rundir='/var/run/frr', vty_socket=None, daemon='', test_reset=False)"
2024-12-07 20:05:26,072  INFO: Loading Config object from file vrf.txt
2024-12-07 20:05:26,079 DEBUG: LINE vrf red                                           : enter context ['vrf red']
2024-12-07 20:05:26,079 DEBUG: LINE ip route 10.0.0.0/8 blackhole 250 tag 210         : add to current context ['vrf red']                                   
2024-12-07 20:05:26,079 DEBUG: LINE exit-vrf                                          : exit context ['vrf red']
2024-12-07 20:05:26,087 DEBUG: New Frr Config
!
vrf red
ip route 10.0.0.0/8 blackhole 250 tag 210
exit-vrf
end
!

end

2024-12-07 20:05:26,087  INFO: Loading Config object from vtysh show running
2024-12-07 20:05:26,097 DEBUG: LINE frr version 10.2                                  : single-line context
2024-12-07 20:05:26,097 DEBUG: LINE frr defaults traditional                          : single-line context
2024-12-07 20:05:26,097 DEBUG: LINE hostname LR1.wue3                                 : single-line context
2024-12-07 20:05:26,097 DEBUG: LINE service integrated-vtysh-config                   : single-line context
2024-12-07 20:05:26,097 DEBUG: LINE vrf red                                           : enter context ['vrf red']
2024-12-07 20:05:26,097 DEBUG: LINE ip route 10.0.0.0/8 blackhole tag 210 250         : add to current context ['vrf red']                                   
2024-12-07 20:05:26,097 DEBUG: LINE exit-vrf                                          : exit context ['vrf red']
2024-12-07 20:05:26,097 DEBUG: Running Frr Config (Pass #0)
frr version 10.2
frr defaults traditional
hostname LR1.wue3
service integrated-vtysh-config
!
vrf red
ip route 10.0.0.0/8 blackhole tag 210 250
exit-vrf
end
!
end
2024-12-07 20:05:26,098  INFO: "frr version 10.2" cannot be removed
2024-12-07 20:05:26,098  INFO: "frr defaults traditional" cannot be removed
2024-12-07 20:05:26,098  INFO: "service integrated-vtysh-config" cannot be removed
2024-12-07 20:05:26,105  INFO: Executed "no hostname LR1.wue3"
2024-12-07 20:05:26,105  INFO: /var/run/frr/reload-PQ0XRB.txt content
['vrf red\n ip route 10.0.0.0/8 blackhole 250 tag 210\nexit\n',
 'vrf red\n no ip route 10.0.0.0/8 blackhole tag 210 250\nexit\n']
[5456|mgmtd] sending configuration
[5457|zebra] sending configuration
[5458|ripd] sending configuration
[5460|ospfd] sending configuration
[5457|zebra] done
[5459|ripngd] sending configuration
[5461|ospf6d] sending configuration
[5462|ldpd] sending configuration
[5460|ospfd] done
[5462|ldpd] done
[5461|ospf6d] done
[5468|babeld] sending configuration
[5470|fabricd] sending configuration
[5471|watchfrr] sending configuration
[5473|staticd] sending configuration
[5463|bgpd] sending configuration
[5459|ripngd] done
[5464|isisd] sending configuration
[5456|mgmtd] done
[5458|ripd] done
[5471|watchfrr] done
[5468|babeld] done
[5470|fabricd] done
[5464|isisd] done
[5473|staticd] done
[5463|bgpd] done
[5474|bfdd] sending configuration
Waiting for children to finish applying config...
[5477|pim6d] sending configuration
[5474|bfdd] done
[5477|pim6d] done
2024-12-07 20:05:26,121  INFO: Loading Config object from vtysh show running
2024-12-07 20:05:26,130 DEBUG: LINE frr version 10.2                                  : single-line context
2024-12-07 20:05:26,130 DEBUG: LINE frr defaults traditional                          : single-line context
2024-12-07 20:05:26,130 DEBUG: LINE hostname LR1.wue3                                 : single-line context
2024-12-07 20:05:26,130 DEBUG: LINE service integrated-vtysh-config                   : single-line context
2024-12-07 20:05:26,130 DEBUG: LINE vrf red                                           : enter context ['vrf red']
2024-12-07 20:05:26,130 DEBUG: LINE exit-vrf                                          : exit context ['vrf red']
2024-12-07 20:05:26,130 DEBUG: Running Frr Config (Pass #1)
frr version 10.2
frr defaults traditional
hostname LR1.wue3
service integrated-vtysh-config
!
vrf red
exit-vrf
end
!
end
2024-12-07 20:05:26,130  INFO: "frr version 10.2" cannot be removed
2024-12-07 20:05:26,130  INFO: "frr defaults traditional" cannot be removed
2024-12-07 20:05:26,130  INFO: "service integrated-vtysh-config" cannot be removed
2024-12-07 20:05:26,130  INFO: /var/run/frr/reload-MPL2TK.txt content
['vrf red\n ip route 10.0.0.0/8 blackhole 250 tag 210\nexit\n',
 'vrf red\n ip route 10.0.0.0/8 blackhole 250 tag 210\nexit\n',
 'vrf red\n no ip route 10.0.0.0/8 blackhole tag 210 250\nexit\n']
[5481|mgmtd] sending configuration
[5483|ripd] sending configuration
[5482|zebra] sending configuration
[5493|babeld] sending configuration
[5484|ripngd] sending configuration
[5483|ripd] done
[5485|ospfd] sending configuration
[5486|ospf6d] sending configuration
[5487|ldpd] sending configuration
[5488|bgpd] sending configuration
[5489|isisd] sending configuration
MGMTD: No changes found to be committed!
[5481|mgmtd] done
[5485|ospfd] done
[5484|ripngd] done
[5487|ldpd] done
[5493|babeld] done
[5495|fabricd] sending configuration
[5482|zebra] done
[5486|ospf6d] done
[5489|isisd] done
[5488|bgpd] done
[5496|watchfrr] sending configuration
[5498|staticd] sending configuration
Waiting for children to finish applying config...
[5502|pim6d] sending configuration
[5499|bfdd] sending configuration
[5495|fabricd] done
[5498|staticd] done
[5496|watchfrr] done
[5499|bfdd] done
[5502|pim6d] done

Expected behavior

vtysh configuration should be the same on every config reload

Actual behavior

routing inconsitencies between individual frr-reload calls

Additional context

Changing the order of the reload configuration to

!
vrf red
 ip route 10.0.0.0/8 blackhole tag 210 250
exit-vrf
!

Will not remove the config

Checklist

• I have searched the open issues for this bug.
• I have not included sensitive information in this report.

[ykholod]

Here is the problem, these two instructions are exactly the same from the point of view of the vtysh:

pppoe-server(config-vrf)# ip route 10.0.0.0/8 blackhole tag 220 250
pppoe-server(config-vrf)# ip route 10.0.0.0/8 blackhole 250 tag 220

When you apply config frr-reload.py forms a temporary config to apply. It consists of two parts: one is to apply newly added instructions (first pass) and one to remove unnecessary instructions (second pass). In order to create temp config, frr-reload.py reads running config and applied config, and compares them.

The trick is that this instruction in running and applied config is written in different way. So, we apply the new one (from the config) and remove the old one (from the running config). But they are the same, so, at-the-end-of-a-day, we just remove it.
Also, this explains, why modifying applied config instruction helps. That's a nice workaround.