Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scheduler: Debounce commit events #1287

Merged
merged 1 commit into from
Aug 1, 2016

Conversation

aaronlehmann
Copy link
Collaborator

When loading a state that contained large numbers of nodes and tasks,
but no ready nodes that could accept the tasks, swarmd used large
amounts of CPU repeatedly trying to schedule the full set of tasks. The
allocator caused many commits on startup (see #1286), and this produced
a large backlog of commit events, each one of which caused a full
scheduling pass.

To avoid this pathological behavior, debounce the commit events
similarly to how the dispatcher's Tasks loop debounces events. When a
commit event is received, that starts a 50 ms countdown to wait for
another commit event before running the scheduling pass. If commit
events keep being received and resetting this timer, the scheduler will
run the scheduling pass anyway after a second.

I think we should consider this for 1.12.1. Without this change, the scheduler can consume 100% CPU for a very long time trying to catch up with the event backlog.

cc @dongluochen

When loading a state that contained large numbers of nodes and tasks,
but no ready nodes that could accept the tasks, swarmd used large
amounts of CPU repeatedly trying to schedule the full set of tasks. The
allocator caused many commits on startup (see moby#1286), and this produced
a large backlog of commit events, each one of which caused a full
scheduling pass.

To avoid this pathological behavior, debounce the commit events
similarly to how the dispatcher's Tasks loop debounces events. When a
commit event is received, that starts a 50 ms countdown to wait for
another commit event before running the scheduling pass. If commit
events keep being received and resetting this timer, the scheduler will
run the scheduling pass anyway after a second.

Signed-off-by: Aaron Lehmann <[email protected]>
@codecov-io
Copy link

Current coverage is 54.95% (diff: 74.19%)

Merging #1287 into master will increase coverage by 0.02%

@@             master      #1287   diff @@
==========================================
  Files            78         78          
  Lines         12428      12454    +26   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6826       6844    +18   
- Misses         4668       4672     +4   
- Partials        934        938     +4   

Sunburst

Powered by Codecov. Last update 4e27e1b...77c62db

@dongluochen
Copy link
Contributor

LGTM

@aaronlehmann aaronlehmann added this to the 1.12.1 milestone Aug 1, 2016
@aaronlehmann aaronlehmann merged commit 008c7f6 into moby:master Aug 1, 2016
@aaronlehmann aaronlehmann deleted the scheduler-debouncing branch August 1, 2016 23:39
@stevvooe
Copy link
Contributor

stevvooe commented Aug 1, 2016

In the future, we should try to use https://godoc.org/golang.org/x/time/rate for rate limiting.

@aaronlehmann
Copy link
Collaborator Author

I filed #1290 about generalizing the rate limiting. Thanks for the pointer - I'll look into this when I get to working on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants