ci(reborn): retry crates.io network failures in the closure (CARGO_NET_RETRY) - #5115
Merged
Merged
Conversation
…T_RETRY) The 64-crate closure runs ~64 jobs that each resolve+download the dependency tree from crates.io in parallel. A transient registry hiccup (seen twice: 'download of once_cell_polyfill failed' HTTP2 framing, and 'agent-client-protocol ... curl [55] SSL_ERROR_SYSCALL') hits many jobs at once and reddens the whole run even though nothing is wrong with the code. Bump Cargo's network retry from the default 3 to 10 and fetch git deps via the git CLI, so transient download failures self-heal instead of failing the job. The heavier fix (build-once `nextest archive` + shard the run, eliminating 64x dependency resolution) remains a separate follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
railway-app
Bot
temporarily deployed
to
ironclaw-ci-preview / ironclaw-pr-5115
June 21, 2026 20:54
Destroyed
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughSummary by CodeRabbit
WalkthroughTwo environment variables — ChangesReborn CI Network Resilience
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
github-actions
Bot
added
size: XS
|
🚅 Deployed to the ironclaw-pr-5115 environment in ironclaw-ci-preview
|
serrrfirat
added a commit
that referenced
this pull request
Jun 21, 2026
…r-crate caches (#5118) The closure's crate-tests matrix used a per-crate cache key (`reborn-tests-<crate>`), producing ~60 caches that each bundle the cargo registry + a target dir (~0.4-0.7 GB each), plus 4 per-partition root caches (~1 GB each). That's ~30+ GB competing for GitHub's ~10 GB per-repo cache LRU, so the caches evict each other almost as fast as they're written: a live snapshot showed only 10 of 64 crates had a cache present, summing to 18 GB and actively evicting. The result is "No cache found" on most jobs -> the crates.io registry is re-downloaded from cold every run, and 64 parallel cold downloads amplify transient registry flakes (the SSL_ERROR_SYSCALL / HTTP2 reds we saw). Switch both matrices to `shared-key`: - crate-tests -> `shared-key: reborn-tests-crates` (one entry for all crate jobs; the registry + shared-dep build is downloaded/compiled once and resident, not ~60x). - root partitions -> `shared-key: reborn-tests-root` (all 4 partitions compile the identical root build, so one cache is strictly better than 4 copies). This drops the reborn-tests cache footprint from ~30+ GB / 68 entries to ~2 entries that comfortably fit the LRU, so the registry stays resident and stops being re-downloaded. Complements CARGO_NET_RETRY (#5115), which remains the safety net for the now-rare cold download. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 22, 2026
This was referenced Jun 26, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 64-crate closure (#5110) runs ~64 jobs that each resolve+download the dependency tree from crates.io in parallel. A transient registry SSL/HTTP2 hiccup hits many jobs at once and reddens the whole run — seen twice already:
download of once_cell_polyfill failed(HTTP2 framing)agent-client-protocol ... curl [55] OpenSSL SSL_ERROR_SYSCALL(61 jobs red in one run)Neither is a code failure — they pass on re-run.
Fix
Add to
reborn-tests.ymlenv:CARGO_NET_RETRY: "10"— Cargo retries network failures (default 3) far more aggressively, so transient download errors self-heal within the job.CARGO_NET_GIT_FETCH_WITH_CLI: "true"— use the git CLI for git deps (more robust than libgit2 under flaky networks).2-line env change, applies to all reborn-tests jobs. Zero behavior change otherwise.
Follow-up (not this PR)
The real scaling fix is build-once
cargo nextest archive→ shard the run, which eliminates the 64× parallel dependency resolution entirely. Tracked for the bake phase.Automated agent-authored.