fix(cron): stagger missed jobs on restart to prevent gateway overload - #18925
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the gateway restarts with many overdue cron jobs, they are now executed with staggered delays to prevent overwhelming the gateway.
Fixes #18892
Summary
Describe the problem and fix in 2–5 bullets:
Change Type (select all)
Scope (select all touched areas)
Linked Issue/PR
User-visible / Behavior Changes
List user-visible changes (including defaults/config).
If none, write
None.Security Impact (required)
Yes/No)Yes/No)Yes/No)Yes/No)Yes/No)Yes, explain risk + mitigation:Repro + Verification
Environment
Steps
Expected
Actual
Evidence
Attach at least one:
Human Verification (required)
What you personally verified (not just CI), and how:
Compatibility / Migration
Yes/No)Yes/No)Yes/No)Failure Recovery (if this breaks)
Risks and Mitigations
List only real risks for this PR. Add/remove entries as needed. If none, write
None.Greptile Summary
This PR addresses #18892 by staggering the execution of missed cron jobs when the gateway restarts with many overdue jobs. Previously, all missed jobs fired simultaneously, potentially overwhelming the gateway.
missedJobStaggerMs(default 5s) andmaxMissedJobsPerRestart(default 5) as configurable options onCronServiceDepsnextRunAtMs(most overdue first), splits into immediate and deferred batchesnextRunAtMsvalues picked up by the normal timerlocked()mechanism andrecomputeNextRuns(which preserves futurenextRunAtMsvalues)staggerMsinstead ofmaxImmediate * staggerMs, so the first deferred jobs will be past-due by the timearmTimerruns (after the immediate batch completes), causing them to fire back-to-back without the intended staggeringConfidence Score: 3/5
staggerMsfromnow), meaning the first several deferred jobs will be past-due by the timearmTimerruns after the immediate batch completes, causing them to fire back-to-back and partially defeating the stagger purpose. The fix is straightforward (start offset atmaxImmediate * staggerMs).src/cron/service/timer.ts— the deferred job stagger offset calculation inrunMissedJobsneeds adjustment to account for immediate batch execution time.Last reviewed commit: eabeee0
(3/5) Reply to the agent's comments like "Can you suggest a fix for this @greptileai?" or ask follow-up questions!