orchestrator/update: Only shut down old tasks on success #2308
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In start-then-stop mode, the updater waits for a verdict on the startup of a new task, and then shuts down the tasks that this new task was meant to replace. It should really only shut down the old task if the new task successfully reached running. The thinking here was probably that a task which fails on startup would briefly enter the running state before failing, so there isn't much point in distinguishing between running and some terminal state. However, if a health check is being used, the task may never enter the running state. This can also happen when a task is rejected by a node. Signed-off-by: Aaron Lehmann <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #2308 +/- ##
==========================================
+ Coverage 61.07% 61.15% +0.08%
==========================================
Files 128 128
Lines 20556 20556
==========================================
+ Hits 12554 12572 +18
+ Misses 6627 6601 -26
- Partials 1375 1383 +8 |
Contributor
|
Is there any way this will make it into Docker 17.06 EE? We're really looking forward to finally start using Swarm Mode services, but without service interruptions on rollbacks. |
Contributor
|
LGTM |
|
How's this tested? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In start-then-stop mode, the updater waits for a verdict on the startup
of a new task, and then shuts down the tasks that this new task was
meant to replace. It should really only shut down the old task if the
new task successfully reached running. The thinking here was probably
that a task which fails on startup would briefly enter the running state
before failing, so there isn't much point in distinguishing between
running and some terminal state. However, if a health check is being
used, the task may never enter the running state. This can also happen
when a task is rejected by a node.
See moby/moby#34111 for a repro case.
cc @sirlatrom @cpuguy83