Skip to content

fix: widen CSMA post-reset jitter to prevent thundering herd #2304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ralphbean merged 1 commit into from
Jun 16, 2026
+15 −0
Merged

fix: widen CSMA post-reset jitter to prevent thundering herd #2304

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions internal/scaffold/fullsend-repo/scripts/lib/github-api-csma.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
# GITHUB_CSMA_MIN_REMAINING_GRAPHQL — default 100
# GITHUB_CSMA_SLOT_MIN_MS — default 250
# GITHUB_CSMA_SLOT_MAX_MS — default 750 (0 disables jitter)
# GITHUB_CSMA_SPREAD_MAX_SEC — default 60 (post-reset desync spread)
# GITHUB_CSMA_BACKOFF_CAP_SEC — default 120

# shellcheck shell=bash
Expand Down Expand Up @@ -41,6 +42,10 @@ _github_csma_slot_max_ms() {
echo "${GITHUB_CSMA_SLOT_MAX_MS:-750}"
}

_github_csma_spread_max_sec() {
echo "${GITHUB_CSMA_SPREAD_MAX_SEC:-60}"
}

_github_csma_backoff_cap_sec() {
echo "${GITHUB_CSMA_BACKOFF_CAP_SEC:-120}"
}
Expand Down Expand Up @@ -85,6 +90,16 @@ github_csma_sense() {

echo "Rate limit sense: ${resource} remaining=${remaining} (min=${min_remaining}); waiting ${wait_secs}s until reset..." >&2
sleep "${wait_secs}"

# After a rate-limit sleep, all runners wake at the same reset timestamp.
# Spread them over a wide window to avoid a thundering herd.
local spread_max
spread_max=$(_github_csma_spread_max_sec)
if (( spread_max > 0 )); then
local spread_secs=$(( RANDOM % spread_max ))
echo "Rate limit reset — spreading ${spread_secs}s to desync from other runners..." >&2
sleep "${spread_secs}"
fi
}

# Random inter-call delay (slot time) to reduce synchronized collisions.
Expand Down
Loading