Skip to content

fix(github): increase retry attempts and add jitter to backoff - #2764

Merged
ralphbean merged 1 commit into
mainfrom
improve-github-retry-backoff
Jun 30, 2026
Merged

fix(github): increase retry attempts and add jitter to backoff#2764
ralphbean merged 1 commit into
mainfrom
improve-github-retry-backoff

Conversation

@ralphbean

Copy link
Copy Markdown
Member

Summary

  • Increases maxRetries from 3 to 5 in the GitHub API client, giving ~5 minutes of patience for secondary rate limits (60s+ base backoff per attempt)
  • Adds jitter (randomizes delay between 50-100% of base) to prevent thundering-herd effects when parallel callers hit rate limits simultaneously
  • Retry-After header values are still respected exactly (no jitter applied)

Context

E2e tests were failing with 403 retryable error after 3 attempts on POST /orgs/halfsend-06/repos — the client gave up too quickly on secondary rate limits. The shell-side CSMA library (github-api-csma.sh) already uses 8 attempts with jitter; this brings the Go client closer to parity.

Test plan

  • New test: TestDo_MaxRetries5 — verifies 5 attempts before exhaustion
  • New test: TestRetryDelay_HasJitter — verifies non-deterministic delays for 5xx
  • New test: TestRetryDelay_SecondaryRateLimit_HasJitter — verifies jitter on 403 backoff
  • New test: TestRetryDelay_RespectsRetryAfterHeader — verifies Retry-After is exact
  • Updated existing TestCreateOrUpdateFile_MaxRetriesExceeded for new retry count
  • Full go test ./internal/forge/github/ passes

🤖 Generated with Claude Code

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Fix GitHub retry backoff (5 retries + jitter) and URL skill dir naming

🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Increase GitHub API client max retries to better tolerate secondary rate limits.
• Add jittered exponential backoff while honoring Retry-After exactly.
• Fix URL skill resolution to return stable, user-meaningful directory names via symlink.
Diagram

graph TD
A["Forge code"] --> B["GitHub LiveClient.do()"] --> C["retryDelay()"] --> D["GitHub API"]
E["resolveSkillDirURL()"] --> F[("Cache dir")] --> G["Named symlink"] --> H["Harness skills path"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Inject RNG/jitter strategy (testable backoff policy)
  • ➕ Avoids probabilistic tests by allowing deterministic seeded RNG in unit tests
  • ➕ Makes it easy to switch between 'full jitter', 'decorrelated jitter', or no jitter depending on endpoint/error type
  • ➖ Slightly more plumbing (policy or function injection) for a small client
  • ➖ More surface area to maintain if not otherwise needed
2. Use a standard backoff library
  • ➕ Battle-tested jitter algorithms and retry policies
  • ➕ Often includes observability hooks and context cancellation patterns
  • ➖ Adds a dependency for a relatively small amount of logic
  • ➖ May require adapting library behavior to GitHub-specific rules (Retry-After exactness, 403 secondary limits)

Recommendation: Current approach is reasonable and minimal: it extends existing backoff logic, adds jitter only when the server doesn’t specify Retry-After, and increases retries to reduce flaky e2e failures. If jitter-related tests ever become flaky in CI, consider refactoring retryDelay to accept an injected RNG/backoff policy so tests can be deterministic while production remains randomized.

Files changed (4) +109 / -10

Bug fix (2) +32 / -7
github.goIncrease retries and add jittered backoff (honor Retry-After) +15/-7

Increase retries and add jittered backoff (honor Retry-After)

• Raises the GitHub client retry cap from 3 to 5. Updates retryDelay to compute a base backoff (including a longer 403 secondary-rate-limit backoff) and applies 50–100% jitter unless a valid Retry-After header is present.

internal/forge/github/github.go

resolve.goReturn URL skill directories with stable URL-derived basenames +17/-0

Return URL skill directories with stable URL-derived basenames

• Fixes URL-based skill resolution by creating an idempotent symlink named after the URL directory alongside the cache’s internal tree/ directory. Ensures downstream consumers see the correct skill name and avoids collisions when multiple URL skills are used.

internal/resolve/resolve.go

Tests (2) +77 / -3
github_test.goAdd retry/jitter tests and update max-retry expectations +67/-2

Add retry/jitter tests and update max-retry expectations

• Updates existing tests to expect 5 attempts instead of 3. Adds coverage for retry exhaustion at 5 attempts, jitter variability for 5xx and 403 secondary rate limits, and exact handling of Retry-After without jitter.

internal/forge/github/github_test.go

resolve_test.goAssert URL skill basenames match URL path names (not cache 'tree') +10/-1

Assert URL skill basenames match URL path names (not cache 'tree')

• Strengthens tests to validate that resolved URL skill paths are directories whose basenames match the URL directory name, including cache-hit and multi-skill scenarios. Prevents regressions where all URL skills resolve to a generic tree directory name.

internal/resolve/resolve_test.go

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

Site preview

Preview: https://40c3f7be-site.fullsend-ai.workers.dev

Commit: 4e3bc5af55f069ce1bfd01c15b1d97d1c8ed1bae

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 29, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:25 PM UTC · Completed 7:38 PM UTC
Commit: f6374ba · View workflow run →

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 51 rules

Grey Divider


Action required

1. Symlink path escape 🐞 Bug ⛨ Security
Description
resolveSkillDirURL() uses filepath.Base(forgeInfo.Path) as a filesystem name without rejecting "..",
so a URL path ending in "/.." makes namedPath resolve to the parent of the cache directory. This can
create/overwrite a symlink outside the cache root and then return that escaped path as the resolved
skill directory.
Code

internal/resolve/resolve.go[R374-387]

+	skillName := filepath.Base(forgeInfo.Path)
+	if skillName == "" || skillName == "." {
+		skillName = "tree"
+	}
+	namedPath := filepath.Join(filepath.Dir(treePath), skillName)
+	if namedPath != treePath {
+		// Idempotent: only create if it doesn't already exist.
+		if _, err := os.Lstat(namedPath); os.IsNotExist(err) {
+			if err := os.Symlink("tree", namedPath); err != nil {
+				return Dependency{}, "", fmt.Errorf("creating named symlink for %s: %w", field, err)
+			}
+		}
+		treePath = namedPath
+	}
Relevance

⭐⭐⭐ High

Security/filesystem hardening commonly accepted; resolve safety concerns raised in #1623 and major
resolve changes in #2139.

PR-#1623
PR-#2139

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
ForgeURLInfo.Path is built from raw URL segments, so it can be ".."; resolveSkillDirURL then uses
filepath.Base(Path) to form a joined filesystem path and creates a symlink there.

internal/forge/url.go[48-93]
internal/resolve/resolve.go[372-388]
PR-#1177

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`resolveSkillDirURL()` derives `skillName := filepath.Base(forgeInfo.Path)` and uses it to build `namedPath := filepath.Join(filepath.Dir(treePath), skillName)`. If `forgeInfo.Path` is `".."` (possible because forge URL parsing does not sanitize path segments), then `namedPath` escapes to the parent directory and `os.Symlink("tree", namedPath)` attempts to create a link outside the cache directory.

### Issue Context
`forge.ParseForgeURL()` constructs `ForgeURLInfo.Path` directly from URL segments after `{ref}` without filtering `.`/`..` segments. The new symlink creation runs even on cache hits (no network call required).

### Fix Focus Areas
- internal/resolve/resolve.go[372-388]
- internal/forge/url.go[48-92]

### Implementation notes
- Validate the derived `skillName` before using it:
 - Reject `".."` (and keep existing `""`/"." handling).
 - Optionally reject any path that contains `".."` segments at all (safer).
- Additionally enforce containment: after computing `namedPath`, ensure it stays within `filepath.Dir(treePath)` (or within the cache root) using `filepath.Rel`/prefix checks on cleaned absolute paths.
- If invalid, skip creating the named symlink and keep `treePath` pointing at `<cache>/tree`.
- Add a unit test that uses a forge URL ending with `/tree/<ref>/..` (with a pre-populated cache entry) and asserts resolution does not write outside the cache directory.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Backoff minimum violated 🐞 Bug ☼ Reliability
Description
retryDelay() adds jitter by returning base/2..base, which can make 403 secondary-rate-limit delays
shorter than secondaryRateLimitBackoff despite it being documented as a minimum. This can cause
premature retries while still rate-limited.
Code

internal/forge/github/github.go[R233-236]

+	// Add jitter: randomize between 50-100% of base to desynchronize
+	// concurrent callers (e.g. parallel e2e test runners).
+	half := base / 2
+	return half + time.Duration(rand.Int64N(int64(half)+1))
Relevance

⭐⭐ Medium

No precedent for “jitter must preserve minimum”; only general retry hardening accepted in
#1612/#2342.

PR-#1612
PR-#2342

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The code explicitly states secondaryRateLimitBackoff is a minimum, but the jitter implementation
halves the computed base duration, making the effective delay less than that minimum for typical
attempts.

internal/forge/github/github.go[211-237]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`retryDelay()` documents `secondaryRateLimitBackoff` as a *minimum* for secondary rate limits, but the new jitter implementation returns `[base/2, base]`. Since `base = secondaryRateLimitBackoff + exp`, halving can produce delays < `secondaryRateLimitBackoff` (e.g., attempt=0: base≈61s → min≈30.5s).

### Issue Context
This behavior is specific to the new jitter logic; `Retry-After` handling remains exact.

### Fix Focus Areas
- internal/forge/github/github.go[211-237]
- internal/forge/github/github_test.go[1831-1861]

### Implementation notes
- Keep `secondaryRateLimitBackoff` as a hard floor for 403-without-Retry-After.
- Apply jitter only to the exponential component (or change jitter range to be >= base), e.g.:
 - `exp := 2^attempt * time.Second`
 - For 403: `delay := secondaryRateLimitBackoff + jitter(exp)` where `jitter(exp)` is `[exp/2, exp]`.
- Add/adjust a test asserting `retryDelay(403, attempt) >= secondaryRateLimitBackoff` (when `Retry-After` absent).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Existing symlink not checked 🐞 Bug ☼ Reliability
Description
If the namedPath already exists, resolveSkillDirURL() still sets treePath=namedPath without
verifying it is a symlink to the cached "tree" directory. This can cause confusing failures or
incorrect behavior if the cache directory contains a stale/corrupted entry at that path.
Code

internal/resolve/resolve.go[R378-387]

+	namedPath := filepath.Join(filepath.Dir(treePath), skillName)
+	if namedPath != treePath {
+		// Idempotent: only create if it doesn't already exist.
+		if _, err := os.Lstat(namedPath); os.IsNotExist(err) {
+			if err := os.Symlink("tree", namedPath); err != nil {
+				return Dependency{}, "", fmt.Errorf("creating named symlink for %s: %w", field, err)
+			}
+		}
+		treePath = namedPath
+	}
Relevance

⭐⭐ Medium

No prior review history about validating existing cache symlink target; closest resolve feedback in
#1623 was undetermined.

PR-#1623

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The current logic only creates the symlink if missing; otherwise it blindly uses the existing path
as the resolved treePath, which is later used as the directory root for reading SKILL.md.

internal/resolve/resolve.go[372-388]
internal/resolve/resolve.go[424-438]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
When `namedPath` already exists, the code does not validate what it is (file/dir/symlink target) and still assigns `treePath = namedPath`. Downstream code treats `treePath` as the skill directory root.

### Issue Context
This can happen on repeated runs, partial cache cleanup, or if a previous run created the path with different contents.

### Fix Focus Areas
- internal/resolve/resolve.go[378-387]
- internal/resolve/resolve.go[403-438]

### Implementation notes
- If `os.Lstat(namedPath)` succeeds:
 - If it is a symlink, `os.Readlink` and require it points to `"tree"` (or resolve and require it equals the intended `treePath`).
 - If it is not a symlink (or points elsewhere), either:
   - return an error indicating cache corruption, or
   - ignore `namedPath` and keep using `treePath`.
- Consider making the behavior explicit in logs/errors to aid debugging of cache issues.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment thread internal/resolve/resolve.go Outdated
Comment on lines +374 to +387
skillName := filepath.Base(forgeInfo.Path)
if skillName == "" || skillName == "." {
skillName = "tree"
}
namedPath := filepath.Join(filepath.Dir(treePath), skillName)
if namedPath != treePath {
// Idempotent: only create if it doesn't already exist.
if _, err := os.Lstat(namedPath); os.IsNotExist(err) {
if err := os.Symlink("tree", namedPath); err != nil {
return Dependency{}, "", fmt.Errorf("creating named symlink for %s: %w", field, err)
}
}
treePath = namedPath
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Symlink path escape 🐞 Bug ⛨ Security

resolveSkillDirURL() uses filepath.Base(forgeInfo.Path) as a filesystem name without rejecting "..",
so a URL path ending in "/.." makes namedPath resolve to the parent of the cache directory. This can
create/overwrite a symlink outside the cache root and then return that escaped path as the resolved
skill directory.
Agent Prompt
### Issue description
`resolveSkillDirURL()` derives `skillName := filepath.Base(forgeInfo.Path)` and uses it to build `namedPath := filepath.Join(filepath.Dir(treePath), skillName)`. If `forgeInfo.Path` is `".."` (possible because forge URL parsing does not sanitize path segments), then `namedPath` escapes to the parent directory and `os.Symlink("tree", namedPath)` attempts to create a link outside the cache directory.

### Issue Context
`forge.ParseForgeURL()` constructs `ForgeURLInfo.Path` directly from URL segments after `{ref}` without filtering `.`/`..` segments. The new symlink creation runs even on cache hits (no network call required).

### Fix Focus Areas
- internal/resolve/resolve.go[372-388]
- internal/forge/url.go[48-92]

### Implementation notes
- Validate the derived `skillName` before using it:
  - Reject `".."` (and keep existing `""`/"." handling).
  - Optionally reject any path that contains `".."` segments at all (safer).
- Additionally enforce containment: after computing `namedPath`, ensure it stays within `filepath.Dir(treePath)` (or within the cache root) using `filepath.Rel`/prefix checks on cleaned absolute paths.
- If invalid, skip creating the named symlink and keep `treePath` pointing at `<cache>/tree`.
- Add a unit test that uses a forge URL ending with `/tree/<ref>/..` (with a pre-populated cache entry) and asserts resolution does not write outside the cache directory.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 29, 2026

Copy link
Copy Markdown

Review

Findings

Low

  • [missing-authorization] internal/forge/github/github.go — This PR lacks a linked issue. Non-trivial changes (20+ changed lines with structural modifications to retry logic) benefit from explicit issue authorization. The PR body provides good context (e2e tests failing with 403 rate limit errors), but a tracking issue would formalize the authorization.
    Remediation: File an issue documenting the e2e test rate-limit failures and link it to this PR.
Previous run

Review

Findings

High

  • [scope-creep] internal/resolve/resolve.go:369 — The PR title claims this is "fix(github): increase retry attempts and add jitter to backoff" but the PR also includes unrelated structural changes to internal/resolve/resolve.go that create named symlinks for skills. The PR description mentions nothing about resolve.go or symlink changes. These are two independent features bundled under a single fix.
    Remediation: Split into two separate PRs, or update the PR title and description to cover both changes.

Medium

  • [race condition / error handling] internal/resolve/resolve.go — The symlink creation has a TOCTOU race and incomplete error handling. Between the os.Lstat(namedPath) check and the os.Symlink call, another process could create the symlink. Additionally, if Lstat returns a non-nil error that is not IsNotExist (e.g., permission denied), the code silently skips symlink creation but still reassigns treePath = namedPath, causing treePath to reference a path that may not exist.
    Remediation: Replace the Lstat+Symlink pair with an unconditional os.Symlink call and ignore os.IsExist errors. This eliminates the TOCTOU race and handles the error path correctly.

Low

  • [missing-authorization] — This PR lacks a linked issue. The PR body provides context (e2e tests failing with 403 rate limit errors), but non-trivial changes across two unrelated packages benefit from explicit issue authorization.
  • [undocumented-design] internal/resolve/resolve.go:369 — The symlink naming logic introduces new behavior with no issue or design context explaining why named symlinks are needed. Inline comments explain the purpose but the broader motivation is undocumented.
  • [commit-prefix] — The PR title uses fix(github): but the changes include a separate symlink feature in resolve.go not covered by the prefix scope.
  • [import-convention] internal/forge/github/github.go:14 — Adds math/rand/v2 for jitter. While math/rand/v2 is appropriate here (non-cryptographic randomness), note that e2e/admin/testutil.go is the only other user — most randomness in the codebase uses crypto/rand for security contexts.
  • [path traversal] internal/resolve/resolve.go:372skillName derived from filepath.Base(forgeInfo.Path) without explicitly rejecting .. as the sole path component. Not currently exploitable (OS rejects .. as a symlink name), but adding skillName == ".." to the guard condition would close the defense-in-depth gap.
  • [code-organization] internal/resolve/resolve.go — The ~12-line symlink creation block is inserted into an already long function. Extracting it into a helper would improve testability and readability.

Labels: PR modifies the GitHub forge client (retry/jitter) and skill resolution caching (symlinks), spanning the harness and skills components.

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Comment thread internal/resolve/resolve.go
}
treePath = filepath.Join(cachePath, "tree")
} else {
fetchedAt = dirEntry.FetchTime

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] undocumented-design

The symlink naming logic introduces new behavior with no issue or design context explaining why named symlinks are needed.

"fmt"
"io"
"math"
"math/rand/v2"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] import-convention

Adds math/rand/v2 for jitter. Appropriate for non-cryptographic randomness but rare in this codebase outside e2e tests.

Comment thread internal/resolve/resolve.go Outdated
@fullsend-ai-review fullsend-ai-review Bot added component/harness Agent harness, config, and skills loading component/skills labels Jun 29, 2026
The GitHub API client was giving up after only 3 retries with
deterministic backoff delays. When multiple e2e test runners hit
secondary rate limits simultaneously, the identical retry timing
caused thundering-herd effects that prolonged rate limiting.

- Increase maxRetries from 3 to 5, giving ~5 minutes of patience
  for secondary rate limits (which use 60s+ base backoff)
- Add jitter (50-100% of base delay) to desynchronize concurrent
  callers, matching the approach in github-api-csma.sh
- Retry-After header values are still used exactly (no jitter)

Signed-off-by: Robin Bender Ginn <rbenderg@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 29, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:55 PM UTC · Completed 8:06 PM UTC
Commit: 4e3bc5a · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jun 29, 2026
@ralphbean
ralphbean added this pull request to the merge queue Jun 30, 2026
Merged via the queue into main with commit 0851b08 Jun 30, 2026
21 checks passed
@ralphbean
ralphbean deleted the improve-github-retry-backoff branch June 30, 2026 11:57
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jun 30, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 12:01 PM UTC · Completed 12:10 PM UTC
Commit: 4e3bc5a · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #2764fix(github): increase retry attempts and add jitter to backoff

Verdict: Workflow performed well. No new proposals — existing issues cover the one identified gap.

Timeline

  1. 19:22 UTC — PR opened by Ralph Bean. Single commit adding jitter to GitHub API retry backoff and increasing maxRetries from 3→5. The initial PR also included unrelated changes to internal/resolve/resolve.go (symlink naming for URL-based skills).
  2. 19:25–19:38 — First review agent run → CHANGES_REQUESTED. Correctly identified scope-creep (high): PR title claims retry fix but includes unrelated resolve.go changes. Also flagged TOCTOU race, path traversal, missing-authorization, and import-convention issues.
  3. 19:28 — Qodo bot posted 3 findings, including a potential correctness issue: jitter can make 403 secondary-rate-limit delays shorter than the documented minimum (secondaryRateLimitBackoff).
  4. ~19:52 — Author pushed an update removing the resolve.go changes, narrowing the PR to just the retry/backoff fix.
  5. 19:55–20:06 — Second review agent run → APPROVED. Only remaining finding: low-severity missing-authorization (no linked issue).
  6. 09:05 UTC next day — Human reviewer (rh-hemartin) approved.
  7. PR merged.

What went well

  • Scope-creep detection was accurate and actionable. The review agent correctly identified that resolve.go changes were unrelated to the retry fix. The author responded by splitting them out — exactly the right outcome.
  • Single rework cycle. One CHANGES_REQUESTED → one fix → APPROVED. Efficient iteration.
  • Verdict stability. The second review correctly dropped all resolve.go findings (removed code) and approved the focused retry/backoff change.
  • Test coverage. Four new tests were added; Codecov confirmed full patch coverage.

One observation (already tracked)

Qodo flagged a potential correctness issue in the remaining retry code — the jitter implementation (base/2..base) can produce delays below secondaryRateLimitBackoff when that value is intended as a minimum. The review agent didn't surface this on either pass, and the human reviewer also approved without addressing it.

This pattern (another review bot catching something the fullsend review agent missed) is already tracked by #664 ("Review agent: incorporate existing human and third-party bot reviews into assessment") and has a concrete precedent in #1444 ("Review agent missed state-alignment bug caught by CodeRabbit"). No new proposal needed.

Token cost note

The fullsend workflow was triggered 4 times by pull_request_review events (from Qodo, fullsend-review CHANGES_REQUESTED, fullsend-review APPROVED, and human APPROVED). Some of these may have dispatched unnecessary review runs. This is partially addressed by #1125 ("Filter pull_request_review trigger to only fire on changes_requested").

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/harness Agent harness, config, and skills loading component/skills ready-for-merge All reviewers approved — ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants