fix(ci): bun install cache on local $HOME not slow /tmp — real deploy-hang fix (#10839) - #11268
Merged
Merged
Conversation
…al deploy-hang root cause (#10839) The canary->latest pin (#11235) did NOT fix the deploy hang: run 28572046433 hung on install for 17min+ on a released bun too. Root cause is the install cache living under $PWD (= the checkout dir on the box /tmp), whose contended/slow filesystem stalls bun linking past the 900s cap. bun logs "Slow filesystem detected ... consider setting $BUN_INSTALL_CACHE_DIR to a local folder". Point the cache at $HOME (local disk, persistent/warm) on all three deploy jobs.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
lalalune
added a commit
that referenced
this pull request
Jul 2, 2026
…bbered by #11271 (#10839) (#11298) The refund refactor #11271 (5b714c7) was branched off develop before the two real deploy-hang fixes landed and its squash-merge silently reverted them across all three deploy jobs (deploy-api / deploy-console / deploy-app): - #11235: bun-version canary -> latest (canary's link phase hangs on the self-hosted robot fleet, stalling past the 900s cap x3). - #11268: bun install cache $PWD/.bun-install-cache (under the box's slow, contended /tmp checkout dir) -> $HOME/.bun-install-cache-deploy on local disk. This is the real root cause of the "Slow filesystem detected" bun link-phase timeouts that stranded prod deploys. Failing run 28569263716 (headSha f7326ef) reproduced exactly this: three bun install attempts each timed out >900s in the link phase with "Slow filesystem detected ... /tmp/eliza-checkout-*/.bun-install-cache". After #11271 clobbered the fix, develop HEAD carries the regression again. Restore the file to its 7bcb749 state (the only intervening commit that touched it was the #11271 clobber), re-applying both fixes. The per-ref serialization for the robot-fleet thrash (#11108 / #11188, cancel-in-progress:false) is untouched and remains in place. Refs #10839, #11108, #11235, #11268. Co-authored-by: Shaw <shawgotbags@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…ploy box (#10839) Stopgap while the self-hosted robot box IO is investigated: 3 deploys of the money-integrity wave hung on bun install at the 900s per-attempt cap (both after pinning bun off canary #11235 and moving the cache to local $HOME #11268). Give a single attempt 2400s so a slow-but-completing install can finish instead of being killed mid-link and restarted worse. If it still fails at 2400s, the box needs on-host attention (confirmed true hang).
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…ploy box (#10839) (#11304) Stopgap while the self-hosted robot box IO is investigated: 3 deploys of the money-integrity wave hung on bun install at the 900s per-attempt cap (both after pinning bun off canary #11235 and moving the cache to local $HOME #11268). Give a single attempt 2400s so a slow-but-completing install can finish instead of being killed mid-link and restarted worse. If it still fails at 2400s, the box needs on-host attention (confirmed true hang).
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…2400s install timeout (#10839) Prod deploys run from main (production environment allows only the main branch), but main cloud-cf-deploy still had the install-hang config: bun canary + install cache under $PWD (box /tmp, slow FS) + 900s per-attempt cap. Every money-integrity-wave prod deploy hung on bun install. Ports the three fixes already merged to develop (#11235/#11268/#11304): pin bun to latest, put the install cache on local $HOME (persistent/warm), and raise the per-attempt timeout to 2400s so the slow-but-completing install finishes. With this on main, a main-push deploy installs, then migrate-db runs (branch=main satisfies the production environment policy) pending the required-reviewer approval, then Worker/Pages deploy. Ships the merged money fixes (#11190 payout gate, #11189 cron, escrow, #11163 launch gate) to prod.
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…2400s install timeout (#10839) (#11314) Prod deploys run from main (production environment allows only the main branch), but main cloud-cf-deploy still had the install-hang config: bun canary + install cache under $PWD (box /tmp, slow FS) + 900s per-attempt cap. Every money-integrity-wave prod deploy hung on bun install. Ports the three fixes already merged to develop (#11235/#11268/#11304): pin bun to latest, put the install cache on local $HOME (persistent/warm), and raise the per-attempt timeout to 2400s so the slow-but-completing install finishes. With this on main, a main-push deploy installs, then migrate-db runs (branch=main satisfies the production environment policy) pending the required-reviewer approval, then Worker/Pages deploy. Ships the merged money fixes (#11190 payout gate, #11189 cron, escrow, #11163 launch gate) to prod.
Contributor
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bun-pin #11235 was the wrong root cause — run 28572046433 hung on install 17min+ on released bun too. Real cause: the install cache under `$PWD` sits on the box /tmp (slow/contended FS), stalling bun's link phase past 900s. bun explicitly warns "Slow filesystem detected ... set $BUN_INSTALL_CACHE_DIR to a local folder". Moves the cache to `$HOME/.bun-install-cache-deploy` (local, warm) on all 3 jobs. Refs #10839.
[cloud-audit]