fix(ci): pin cloud-cf-deploy bun off canary — unblock stranded prod deploys (#10839) - #11235
Merged
Merged
Conversation
…ands prod deploys (#10839) All three deploy jobs used bun-version: canary, whose link phase hangs on the self-hosted runners (install resolves+extracts in ~2min then stalls past the 900s timeout, x3 retries), failing the deploy. This stranded prod deploys including run 28569068598 (the money-integrity wave), so merged money fixes never reached production. Pin to the latest RELEASE, which does not hang. Maintainer can pin a specific version if a floating latest is undesirable.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Member
|
Post-merge audit (root cause confirmed against real CI logs: canary 1.4.0-canary.1+eba370b69 extracts then hangs the link phase past the 900s cap ×3 on the self-hosted fleet — runs 28552075615 + 28569068598, the latter the stranded main money-integrity deploy; pin target 1.3.14 ran the full CI install + all three deploy jobs' hard build gates green locally). One low residual: |
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…al deploy-hang root cause (#10839) The canary->latest pin (#11235) did NOT fix the deploy hang: run 28572046433 hung on install for 17min+ on a released bun too. Root cause is the install cache living under $PWD (= the checkout dir on the box /tmp), whose contended/slow filesystem stalls bun linking past the 900s cap. bun logs "Slow filesystem detected ... consider setting $BUN_INSTALL_CACHE_DIR to a local folder". Point the cache at $HOME (local disk, persistent/warm) on all three deploy jobs.
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…al deploy-hang root cause (#10839) (#11268) The canary->latest pin (#11235) did NOT fix the deploy hang: run 28572046433 hung on install for 17min+ on a released bun too. Root cause is the install cache living under $PWD (= the checkout dir on the box /tmp), whose contended/slow filesystem stalls bun linking past the 900s cap. bun logs "Slow filesystem detected ... consider setting $BUN_INSTALL_CACHE_DIR to a local folder". Point the cache at $HOME (local disk, persistent/warm) on all three deploy jobs.
lalalune
added a commit
that referenced
this pull request
Jul 2, 2026
…bbered by #11271 (#10839) (#11298) The refund refactor #11271 (5b714c7) was branched off develop before the two real deploy-hang fixes landed and its squash-merge silently reverted them across all three deploy jobs (deploy-api / deploy-console / deploy-app): - #11235: bun-version canary -> latest (canary's link phase hangs on the self-hosted robot fleet, stalling past the 900s cap x3). - #11268: bun install cache $PWD/.bun-install-cache (under the box's slow, contended /tmp checkout dir) -> $HOME/.bun-install-cache-deploy on local disk. This is the real root cause of the "Slow filesystem detected" bun link-phase timeouts that stranded prod deploys. Failing run 28569263716 (headSha f7326ef) reproduced exactly this: three bun install attempts each timed out >900s in the link phase with "Slow filesystem detected ... /tmp/eliza-checkout-*/.bun-install-cache". After #11271 clobbered the fix, develop HEAD carries the regression again. Restore the file to its 7bcb749 state (the only intervening commit that touched it was the #11271 clobber), re-applying both fixes. The per-ref serialization for the robot-fleet thrash (#11108 / #11188, cancel-in-progress:false) is untouched and remains in place. Refs #10839, #11108, #11235, #11268. Co-authored-by: Shaw <shawgotbags@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…ploy box (#10839) Stopgap while the self-hosted robot box IO is investigated: 3 deploys of the money-integrity wave hung on bun install at the 900s per-attempt cap (both after pinning bun off canary #11235 and moving the cache to local $HOME #11268). Give a single attempt 2400s so a slow-but-completing install can finish instead of being killed mid-link and restarted worse. If it still fails at 2400s, the box needs on-host attention (confirmed true hang).
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…ploy box (#10839) (#11304) Stopgap while the self-hosted robot box IO is investigated: 3 deploys of the money-integrity wave hung on bun install at the 900s per-attempt cap (both after pinning bun off canary #11235 and moving the cache to local $HOME #11268). Give a single attempt 2400s so a slow-but-completing install can finish instead of being killed mid-link and restarted worse. If it still fails at 2400s, the box needs on-host attention (confirmed true hang).
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…2400s install timeout (#10839) Prod deploys run from main (production environment allows only the main branch), but main cloud-cf-deploy still had the install-hang config: bun canary + install cache under $PWD (box /tmp, slow FS) + 900s per-attempt cap. Every money-integrity-wave prod deploy hung on bun install. Ports the three fixes already merged to develop (#11235/#11268/#11304): pin bun to latest, put the install cache on local $HOME (persistent/warm), and raise the per-attempt timeout to 2400s so the slow-but-completing install finishes. With this on main, a main-push deploy installs, then migrate-db runs (branch=main satisfies the production environment policy) pending the required-reviewer approval, then Worker/Pages deploy. Ships the merged money fixes (#11190 payout gate, #11189 cron, escrow, #11163 launch gate) to prod.
NubsCarson
added a commit
that referenced
this pull request
Jul 2, 2026
…2400s install timeout (#10839) (#11314) Prod deploys run from main (production environment allows only the main branch), but main cloud-cf-deploy still had the install-hang config: bun canary + install cache under $PWD (box /tmp, slow FS) + 900s per-attempt cap. Every money-integrity-wave prod deploy hung on bun install. Ports the three fixes already merged to develop (#11235/#11268/#11304): pin bun to latest, put the install cache on local $HOME (persistent/warm), and raise the per-attempt timeout to 2400s so the slow-but-completing install finishes. With this on main, a main-push deploy installs, then migrate-db runs (branch=main satisfies the production environment policy) pending the required-reviewer approval, then Worker/Pages deploy. Ships the merged money fixes (#11190 payout gate, #11189 cron, escrow, #11163 launch gate) to prod.
This was referenced Jul 2, 2026
lalalune
added a commit
that referenced
this pull request
Jul 2, 2026
…deploy job (#10839) (#11318) #11289's new migrate-db job — the schema gate every deploy now depends on — was the one job left on bun canary after #11235 pinned the rest of this workflow following the link-phase hang that stranded two prod deploys (runs 28552075615, 28569068598). A canary regression in the gate would fail-closed-block every deploy. Co-authored-by: moon <thefutureisaweirdplace@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All 3 cloud-cf-deploy jobs pinned bun to `canary`, whose link phase hangs on the self-hosted runners (install resolves in ~2min, then stalls past the 900s cap x3 → deploy fails). This stranded prod deploys — including the in-flight money-integrity wave deploy `28569068598` (payout gate #11190, cron #11189, escrow, gate #11163), so those merged fixes are NOT live for real users. Pin to `latest` (release); a released bun does not hang. @lalalune can pin a specific version if preferred. Refs #10839.
[cloud-audit]