Skip to content

Commit

Permalink
zebra: Prevent starvation in dplane_thread_loop
Browse files Browse the repository at this point in the history
When removing a large number of routes, the linux kernel can take the
cpu for an extended amount of time, leaving a situation where FRR
detects a starvation event.

r1# sharp install routes 10.0.0.0 nexthop 192.168.44.33 1000000 repeat 10
2024-06-14 12:55:49.365 [NTFY] sharpd: [M7Q4P-46WDR] vty[5]@# sharp install routes 10.0.0.0 nexthop 192.168.44.33 1000000 repeat 10
2024-06-14 12:55:49.365 [DEBG] sharpd: [YP4TQ-01TYK] Inserting 1000000 routes
2024-06-14 12:55:57.256 [DEBG] sharpd: [TPHKD-3NYSB] Installed All Items 7.890085
2024-06-14 12:55:57.256 [DEBG] sharpd: [YJ486-NX5R1] Removing 1000000 routes
2024-06-14 12:56:07.802 [WARN] zebra: [QH9AB-Y4XMZ][EC 100663314] STARVATION: task dplane_thread_loop (634377bc8f9e) ran for 7078ms (cpu time 220ms)
2024-06-14 12:56:25.039 [DEBG] sharpd: [WTN53-GK9Y5] Removed all Items 27.783668
2024-06-14 12:56:25.039 [DEBG] sharpd: [YP4TQ-01TYK] Inserting 1000000 routes
2024-06-14 12:56:32.783 [DEBG] sharpd: [TPHKD-3NYSB] Installed All Items 7.743524
2024-06-14 12:56:32.783 [DEBG] sharpd: [YJ486-NX5R1] Removing 1000000 routes
2024-06-14 12:56:41.447 [WARN] zebra: [QH9AB-Y4XMZ][EC 100663314] STARVATION: task dplane_thread_loop (634377bc8f9e) ran for 5175ms (cpu time 179ms)

Let's modify the loop in dplane_thread_loop such that after a provider
has been run, check to see if the event should yield, if so, stop
and reschedule this for the future.

Signed-off-by: Donald Sharp <[email protected]>
  • Loading branch information
donaldsharp committed Jun 14, 2024
1 parent e8ce036 commit 6faad86
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions zebra/zebra_dplane.c
Original file line number Diff line number Diff line change
Expand Up @@ -7441,6 +7441,11 @@ static void dplane_thread_loop(struct event *event)
zlog_debug("dplane dequeues %d completed work from provider %s",
counter, dplane_provider_get_name(prov));

if (event_should_yield(event)) {
reschedule = true;
break;
}

/* Locate next provider */
prov = dplane_prov_list_next(&zdplane_info.dg_providers, prov);
}
Expand Down

0 comments on commit 6faad86

Please sign in to comment.