Use priority queue instead of queue #167
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erwinvaneyk
changed the title
erwinvaneyk
force-pushed
the
eval-heap
branch
3 times, most recently
from
7a9918c to
e31a466
Compare
- The original implementation was too prone to bugs, and too complicated to use
The resolving timeout (that kills a expression if it takes longer than 100ms) could get stuck on a non-responsive channel. This is likely when the 'vm' was garbage collected before the interrupt occured
This introduce a workqueue structure from k8s client, which support thread safety object adding and polling reading.
ghost
mentioned this pull request
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The old implementation of the evaluation queue used a plain notification queue in the form of a gochannel. For each invocation there would be routinely evaluation 'tasks' added to this queue. In case of a large number of invocations, the incoming number of evaluation tasks could exceed the controller's capacity to consume these tasks, causing the queue to congest and tasks to be dropped. Especially since evaluation tasks were added for invocations that were not active (awaiting dynamic invocations to be completed for example), it could happen that evaluations that were known to result in no action would be added, while evaluations of fully active invocations would be dropped. This eventually would slow down the execution of invocations due to the clutter.
To fix this, I converted the queue into a priority queue. This priority queue has a number of constraints:
(1) every task in there is unique, so pushing a invocation evaluation multiple times in the queue will now result into only one being present; (2) the priority is based on two parameters: first, if an event has taken place for the invocation, and otherwise the time since last evaluation. So unless an event has updated the state of the invocation, each workflow invocation gets equal attention from the controller.