ospfd: Improve OSPF neighbor retransmission list granularity and prec…

…ision

The current OSPF neighbor retransmission operates on a single per-neighbor
periodic timer that sends all LSAs on the list when it expires.
Additionally, since it skips the first retransmission of received LSAs so
that at least the retransmission interval (resulting in a delay of between
the retransmission interval and twice the interval. In environments where
the links are lossy on P2MP networks with "delay-reflood" configured (which
relies on neighbor retransmission in partial meshs), the implementation
is sub-optimal (to say the least).

This commit reimplements OSPF neighbor retransmission as follows:

   1. A new data structure making use the application managed
      linklist.c library functions implements an OSPF temporal
      list where each node includes a timestamp.
   2. The existing neighbor LS retransmission LSDB data structure
      is augmented with a pointer to the list node on the temporal
      list to faciliate O(1) removal when the LSA is acknowledged.
   3. The neighbor LS retransmission timer is set to the expiration
      timer of the LSA at the top of the list.
   4. When the timer expires, LSAs are retransmitted that within
      the window of the current time and a small dela (20 milli-secs).
      The LSAs that are retransmited are given an updated
      retransmissionn time and moved to the end of the list.
   5. Neighbor and interface LSA retransmission counters are added
      to provide insight into the lossiness of the links. However,
      these will increment quickly on non-fully meshed P2MP networks
      with "delay-reflood" configured.
   6. Added a topotest to exercise the implementation on a non-fully
      meshed P2MP network with "delay-reflood" configured. The
      alternative was to use existing mechanisms to instroduce loss
      but these seem less determistic in a topotest.

Signed-off-by: Acee Lindem <[email protected]>

doc/developer/ospf-ls-retrans.rst

@@ -0,0 +1,69 @@
+OSPF Neighor Retransmission List
+================================
+
+Overview
+--------
+
+OSPF neighbor link-state retransmission lists are implemented using
+both a sparse Link State Database (LSDB) and a doubly-linked list.
+Rather than previous per-neighbor periodic timer, a per-neighbor
+timer is set to the expiration time of the next scheduled LSA
+retransmission.
+
+Sparse Link State Database (LSDB)
+---------------------------------
+
+When an explicit or implied acknowledgment is recieved from a
+neighbor in 2-way state or higher, the acknowledge LSA must be
+removed from the neighbor's link state retransmission list. In order
+to do this efficiently, a sparse LSDB is utilized. LSDB entries also
+include a pointer to the corresponding list entry so that it may be
+efficiently removed from the doubly-linked list.
+
+The sparse LSDB is implemented using the OSPF functions is
+ospf_lsdb.[c,h]. OSPF LSDBs are implemented as an array of route
+tables (lib/table.[c,h]). What is unique of the LS Retransmission
+list LSDB is that each entry also has a pointer into the doubly-linked
+list to facilitate fast deletions.
+
+Doubly-Linked List
+------------------
+
+In addition to the sparse LSDB, LSAs on a neighbor LS retransmission
+list are also maintained in a linked-list order chronologically
+with the LSA scheduled for the next retransmission at the head of
+the list.
+
+The doubly-link list is implemented using the dlist macros in
+lib/typesafe.h.
+
+LSA LS Retransmission List Addition
+------------------------------------
+
+When an LSA is added to a neighbor retransmission list, it is
+added to both the sparse LSDB and the doubly-linked list with a pointer
+in the LSDB route-table node to the list entry. The LSA is added to
+the tail of the list with the expiration time set to the current time
+with the retransmission interval added. If the neighbor retransmission
+timer is not set, it is set to expire at the time of the newly added
+LSA.
+
+LSA LS Retransmission List Deletion
+-----------------------------------
+
+When an LSA is deleted from a neighbor retransmission list, it is
+deleted from eboth the sparse LSDB and the doubly-linked list with the
+pointer the LSDB route-table node used to efficiently delete the entry
+from the list. If the LSA at the head of the list was removed, then
+the neighbor retransmission timer is reset to the expiration of the
+LSA at the head of the list or canceled if the list is empty.
+
+Neighbor LS Retransmission List Expiration
+------------------------------------------
+
+When the neighbor retransmission timer expires, the LSA at the top of
+list and any in a configured window (e.g., 50 milliseconds) are
+retransmitted. The LSAs that have been retransmitted are removed from
+the list and readded to the tail of the list with a new expiration time
+which is retransmit-interval seconds in the future.
+

doc/developer/ospf.rst

@@ -8,6 +8,7 @@ OSPFD
    :maxdepth: 2
 
    ospf-api
+   ospf-ls-retrans
    ospf-sr
    cspf
 

doc/user/ospfd.rst

@@ -738,7 +738,17 @@ Interfaces
    retransmitting Database Description and Link State Request packets. The
    default value is 5 seconds.
 
-.. clicmd:: ip ospf transmit-delay (1-65535) [A.B.C.D]
+.. clicmd:: ip ospf retransmit-window (20-1000)
+
+
+   Set number of milliseconds in the window for neighbor LSA retransmission.
+   When a neighbor Link State (LS) retransmission timer expires, LSAs scheduled
+   to be retransmitted within the number of milliseconds configured are
+   retransmitted to the neighbor. Any expiring after the window will be
+   retransmitted the next time the neighbor LS retransmission timer expires.
+   The default is 50 milliseconds.
+
+ .. clicmd:: ip ospf transmit-delay (1-65535) [A.B.C.D]
 
 
    Set number of seconds for InfTransDelay value. LSAs' age should be

lib/libospf.h

@@ -58,6 +58,7 @@ extern "C" {
 #define OSPF_HELLO_DELAY_DEFAULT           10
 #define OSPF_ROUTER_PRIORITY_DEFAULT        1
 #define OSPF_RETRANSMIT_INTERVAL_DEFAULT    5
+#define OSPF_RETRANSMIT_WINDOW_DEFAULT	    50 /* milliseconds */
 #define OSPF_TRANSMIT_DELAY_DEFAULT         1
 #define OSPF_DEFAULT_BANDWIDTH		 10000	/* Mbps */
 

ospfd/ospf_flood.c

@@ -1015,7 +1015,7 @@ void ospf_ls_request_delete(struct ospf_neighbor *nbr, struct ospf_lsa *lsa)
 	ospf_lsdb_delete(&nbr->ls_req, lsa);
 }
 
-/* Remove all LSA from neighbor's ls-requenst list. */
+/* Remove all LSAs from neighbor's ls-request list. */
 void ospf_ls_request_delete_all(struct ospf_neighbor *nbr)
 {
 	ospf_lsa_unlock(&nbr->ls_req_last);
@@ -1061,58 +1061,114 @@ int ospf_ls_retransmit_isempty(struct ospf_neighbor *nbr)
 /* Add LSA to be retransmitted to neighbor's ls-retransmit list. */
 void ospf_ls_retransmit_add(struct ospf_neighbor *nbr, struct ospf_lsa *lsa)
 {
-	struct ospf_lsa *old;
+	struct ospf_lsdb_linked_node *ls_rxmt_node;
+	struct ospf_lsa_list_entry *ls_rxmt_list_entry;
+	struct ospf_lsa *old = NULL;
+	bool rxmt_head_replaced = false;
 
-	old = ospf_ls_retransmit_lookup(nbr, lsa);
+	ls_rxmt_node = ospf_lsdb_linked_lookup(&nbr->ls_rxmt, lsa);
+	if (ls_rxmt_node)
+		old = ls_rxmt_node->info;
 
 	if (ospf_lsa_more_recent(old, lsa) < 0) {
 		if (old) {
 			old->retransmit_counter--;
+			if (ls_rxmt_node->lsa_list_entry ==
+			    ospf_lsa_list_first(&nbr->ls_rxmt_list))
+				rxmt_head_replaced = true;
+			ospf_lsa_list_del(&nbr->ls_rxmt_list,
+					  ls_rxmt_node->lsa_list_entry);
+			XFREE(MTYPE_OSPF_LSA_LIST, ls_rxmt_node->lsa_list_entry);
+			ospf_lsdb_delete(&nbr->ls_rxmt, old);
 			if (IS_DEBUG_OSPF(lsa, LSA_FLOODING))
-				zlog_debug("RXmtL(%lu)--, NBR(%pI4(%s)), LSA[%s]",
+				zlog_debug("RXmtL(%lu) NBR(%pI4(%s)) Old Delete LSA[%s] on Add",
 					   ospf_ls_retransmit_count(nbr),
 					   &nbr->router_id,
 					   ospf_get_name(nbr->oi->ospf),
-					   dump_lsa_key(old));
-			ospf_lsdb_delete(&nbr->ls_rxmt, old);
+					   dump_lsa_key(lsa));
+			ospf_lsa_unlock(&old);
 		}
 		lsa->retransmit_counter++;
+		ls_rxmt_list_entry = XCALLOC(MTYPE_OSPF_LSA_LIST,
+					     sizeof(struct ospf_lsa_list_entry));
+		/*
+		 * Set the LSA retransmission time for the neighbor;
+		 */
+		monotime(&ls_rxmt_list_entry->list_entry_time);
+		ls_rxmt_list_entry->list_entry_time.tv_sec += nbr->v_ls_rxmt;
+
+		/*
+		 * Add the LSA to the neighbor retransmission list.
+		 */
+		ls_rxmt_list_entry->lsa = ospf_lsa_lock(lsa);
+		ospf_lsa_list_add_tail(&nbr->ls_rxmt_list, ls_rxmt_list_entry);
+		ospf_lsdb_add(&nbr->ls_rxmt, lsa);
+
 		/*
-		 * We cannot make use of the newly introduced callback function
-		 * "lsdb->new_lsa_hook" to replace debug output below, just
-		 * because
-		 * it seems no simple and smart way to pass neighbor information
-		 * to
-		 * the common function "ospf_lsdb_add()" -- endo.
+		 * Look up the newly added node and set the list pointer.
 		 */
+		ls_rxmt_node = ospf_lsdb_linked_lookup(&nbr->ls_rxmt, lsa);
+		ls_rxmt_node->lsa_list_entry = ls_rxmt_list_entry;
+
 		if (IS_DEBUG_OSPF(lsa, LSA_FLOODING))
-			zlog_debug("RXmtL(%lu)++, NBR(%pI4(%s)), LSA[%s]",
+			zlog_debug("RXmtL(%lu) NBR(%pI4(%s)) Add LSA[%s] retrans at (%ld/%ld)",
 				   ospf_ls_retransmit_count(nbr),
-				   &nbr->router_id,
-				   ospf_get_name(nbr->oi->ospf),
-				   dump_lsa_key(lsa));
-		ospf_lsdb_add(&nbr->ls_rxmt, lsa);
+				   &nbr->router_id, ospf_get_name(nbr->oi->ospf),
+				   dump_lsa_key(lsa),
+				   (long)ls_rxmt_list_entry->list_entry_time
+					   .tv_sec,
+				   (long)ls_rxmt_list_entry->list_entry_time
+					   .tv_usec);
+		/*
+		 * Reset the neighbor LSA retransmission timer if isn't currently
+		 * running or the LSA at the head of the list was updated.
+		 */
+		if (!nbr->t_ls_rxmt || rxmt_head_replaced)
+			ospf_ls_retransmit_set_timer(nbr);
 	}
 }
 
 /* Remove LSA from neibghbor's ls-retransmit list. */
 void ospf_ls_retransmit_delete(struct ospf_neighbor *nbr, struct ospf_lsa *lsa)
 {
-	if (ospf_ls_retransmit_lookup(nbr, lsa)) {
+	struct ospf_lsdb_linked_node *ls_rxmt_node;
+
+	ls_rxmt_node = ospf_lsdb_linked_lookup(&nbr->ls_rxmt, lsa);
+
+	if (ls_rxmt_node) {
+		bool rxmt_timer_reset;
+
+		if (ls_rxmt_node->lsa_list_entry ==
+		    ospf_lsa_list_first(&nbr->ls_rxmt_list))
+			rxmt_timer_reset = true;
+		else
+			rxmt_timer_reset = false;
+
 		lsa->retransmit_counter--;
-		if (IS_DEBUG_OSPF(lsa, LSA_FLOODING)) /* -- endo. */
-			zlog_debug("RXmtL(%lu)--, NBR(%pI4(%s)), LSA[%s]",
+		ospf_lsa_list_del(&nbr->ls_rxmt_list,
+				  ls_rxmt_node->lsa_list_entry);
+		XFREE(MTYPE_OSPF_LSA_LIST, ls_rxmt_node->lsa_list_entry);
+		ospf_lsdb_delete(&nbr->ls_rxmt, lsa);
+		if (IS_DEBUG_OSPF(lsa, LSA_FLOODING))
+			zlog_debug("RXmtL(%lu) NBR(%pI4(%s)) Delete LSA[%s]",
 				   ospf_ls_retransmit_count(nbr),
-				   &nbr->router_id,
-				   ospf_get_name(nbr->oi->ospf),
+				   &nbr->router_id, ospf_get_name(nbr->oi->ospf),
 				   dump_lsa_key(lsa));
-		ospf_lsdb_delete(&nbr->ls_rxmt, lsa);
+		ospf_lsa_unlock(&lsa);
+
+		/*
+		 * If the LS retransmission entry at the head of the list was
+		 * deleted, reset the timer.
+		 */
+		if (rxmt_timer_reset)
+			ospf_ls_retransmit_set_timer(nbr);
 	}
 }
 
 /* Clear neighbor's ls-retransmit list. */
 void ospf_ls_retransmit_clear(struct ospf_neighbor *nbr)
 {
+	struct ospf_lsa_list_entry *ls_rxmt_list_entry;
 	struct ospf_lsdb *lsdb;
 	int i;
 
@@ -1128,10 +1184,54 @@ void ospf_ls_retransmit_clear(struct ospf_neighbor *nbr)
 				ospf_ls_retransmit_delete(nbr, lsa);
 	}
 
+	frr_each_safe (ospf_lsa_list, &nbr->ls_rxmt_list, ls_rxmt_list_entry) {
+		ospf_lsa_list_del(&nbr->ls_rxmt_list, ls_rxmt_list_entry);
+		ospf_lsa_unlock(&ls_rxmt_list_entry->lsa);
+		XFREE(MTYPE_OSPF_LSA_LIST, ls_rxmt_list_entry);
+	}
+
 	ospf_lsa_unlock(&nbr->ls_req_last);
 	nbr->ls_req_last = NULL;
 }
 
+/*
+ * Set the neighbor's ls-retransmit timer based on the next
+ * LSA retransmit time.
+ */
+void ospf_ls_retransmit_set_timer(struct ospf_neighbor *nbr)
+{
+	struct ospf_lsa_list_entry *ls_rxmt_list_entry;
+
+	if (nbr->t_ls_rxmt)
+		EVENT_OFF(nbr->t_ls_rxmt);
+
+	ls_rxmt_list_entry = ospf_lsa_list_first(&nbr->ls_rxmt_list);
+	if (ls_rxmt_list_entry) {
+		struct timeval current_time, delay;
+		unsigned long delay_milliseconds;
+
+		monotime(&current_time);
+		if (timercmp(&current_time,
+			     &ls_rxmt_list_entry->list_entry_time, >=))
+			delay_milliseconds = 10;
+		else {
+			timersub(&ls_rxmt_list_entry->list_entry_time,
+				 &current_time, &delay);
+			delay_milliseconds = (delay.tv_sec * 1000) +
+					     (delay.tv_usec / 1000);
+		}
+
+		event_add_timer_msec(master, ospf_ls_rxmt_timer, nbr,
+				     delay_milliseconds, &nbr->t_ls_rxmt);
+		if (IS_DEBUG_OSPF(lsa, LSA_FLOODING))
+			zlog_debug("RXmtL(%lu) NBR(%pI4(%s)) retrans timer set in %ld msecs - Head LSA(%s)",
+				   ospf_ls_retransmit_count(nbr),
+				   &nbr->router_id, ospf_get_name(nbr->oi->ospf),
+				   delay_milliseconds,
+				   dump_lsa_key(ls_rxmt_list_entry->lsa));
+	}
+}
+
 /* Lookup LSA from neighbor's ls-retransmit list. */
 struct ospf_lsa *ospf_ls_retransmit_lookup(struct ospf_neighbor *nbr,
 					   struct ospf_lsa *lsa)

ospfd/ospf_flood.h

@@ -7,6 +7,26 @@
 #ifndef _ZEBRA_OSPF_FLOOD_H
 #define _ZEBRA_OSPF_FLOOD_H
 
+/*
+ * OSPF Temporal LSA List
+ */
+PREDECL_DLIST(ospf_lsa_list);
+
+struct ospf_lsa_list_entry {
+	/* Linkage for LSA List */
+	struct ospf_lsa_list_item list_linkage;
+
+	/*
+	 * Time associated with the list entry. For example, for a neigbhor
+	 * link retransmission list, this is the retransmission time.
+	 */
+	struct timeval list_entry_time;
+
+	struct ospf_lsa *lsa;
+};
+
+DECLARE_DLIST(ospf_lsa_list, struct ospf_lsa_list_entry, list_linkage);
+
 extern int ospf_flood(struct ospf *, struct ospf_neighbor *, struct ospf_lsa *,
 		      struct ospf_lsa *);
 extern int ospf_flood_through(struct ospf *, struct ospf_neighbor *,
@@ -36,6 +56,8 @@ extern void ospf_ls_retransmit_add(struct ospf_neighbor *, struct ospf_lsa *);
 extern void ospf_ls_retransmit_delete(struct ospf_neighbor *,
 				      struct ospf_lsa *);
 extern void ospf_ls_retransmit_clear(struct ospf_neighbor *);
+extern void ospf_ls_retransmit_set_timer(struct ospf_neighbor *nbr);
+
 extern struct ospf_lsa *ospf_ls_retransmit_lookup(struct ospf_neighbor *,
 						  struct ospf_lsa *);
 extern void ospf_ls_retransmit_delete_nbr_area(struct ospf_area *,

ospfd/ospf_interface.c

@@ -542,6 +542,7 @@ static struct ospf_if_params *ospf_new_if_params(void)
 	UNSET_IF_PARAM(oip, output_cost_cmd);
 	UNSET_IF_PARAM(oip, transmit_delay);
 	UNSET_IF_PARAM(oip, retransmit_interval);
+	UNSET_IF_PARAM(oip, retransmit_window);
 	UNSET_IF_PARAM(oip, passive_interface);
 	UNSET_IF_PARAM(oip, v_hello);
 	UNSET_IF_PARAM(oip, fast_hello);
@@ -599,6 +600,7 @@ void ospf_free_if_params(struct interface *ifp, struct in_addr addr)
 	if (!OSPF_IF_PARAM_CONFIGURED(oip, output_cost_cmd) &&
 	    !OSPF_IF_PARAM_CONFIGURED(oip, transmit_delay) &&
 	    !OSPF_IF_PARAM_CONFIGURED(oip, retransmit_interval) &&
+	    !OSPF_IF_PARAM_CONFIGURED(oip, retransmit_window) &&
 	    !OSPF_IF_PARAM_CONFIGURED(oip, passive_interface) &&
 	    !OSPF_IF_PARAM_CONFIGURED(oip, v_hello) &&
 	    !OSPF_IF_PARAM_CONFIGURED(oip, fast_hello) &&
@@ -695,6 +697,9 @@ int ospf_if_new_hook(struct interface *ifp)
 	IF_DEF_PARAMS(ifp)->retransmit_interval =
 		OSPF_RETRANSMIT_INTERVAL_DEFAULT;
 
+	SET_IF_PARAM(IF_DEF_PARAMS(ifp), retransmit_window);
+	IF_DEF_PARAMS(ifp)->retransmit_window = OSPF_RETRANSMIT_WINDOW_DEFAULT;
+
 	SET_IF_PARAM(IF_DEF_PARAMS(ifp), priority);
 	IF_DEF_PARAMS(ifp)->priority = OSPF_ROUTER_PRIORITY_DEFAULT;
 

ospfd/ospf_interface.h

@@ -47,6 +47,8 @@ struct ospf_if_params {
 			 output_cost_cmd); /* Command Interface Output Cost */
 	DECLARE_IF_PARAM(uint32_t,
 			 retransmit_interval); /* Retransmission Interval */
+	DECLARE_IF_PARAM(uint32_t,
+			 retransmit_window); /* Retransmission Window */
 	DECLARE_IF_PARAM(uint8_t, passive_interface); /* OSPF Interface is
 							passive: no sending or
 							receiving (no need to
@@ -296,6 +298,7 @@ struct ospf_interface {
 	uint32_t ls_ack_out;   /* LS Ack message output count. */
 	uint32_t discarded;    /* discarded input count by error. */
 	uint32_t state_change; /* Number of status change. */
+	uint32_t ls_rxmt_lsa;  /* Number of LSAs retransmitted. */
 
 	uint32_t full_nbrs;