ci: unpin Node 24.16.0 in the root-path e2e job to fix the Playwright install hang - #28
Merged
blackflame007 merged 1 commit intoJul 28, 2026
Conversation
… install hang `test-server-root-path` last passed 2026-06-13 and was cancelled on every run since 2026-07-04, on both matrix legs, always at the same point: the Chrome-for-Testing zip reaches 100% and `npx playwright install` never returns, burning the full 30m job cap. With DEBUG=pw:install (added in #26) the first post-mitigation run named the step exactly, identically on all four attempts and both legs: pw:install -- download complete, size: 175440843 pw:install SUCCESS downloading Chrome for Testing 145.0.7632.6 pw:install removing existing browser directory if any pw:install extracting archive <no further output; killed by `timeout 75`> So the transfer was never the problem; extraction was. Node 24.16.0 regressed stream destruction (nodejs/node#63487), which deadlocks the yauzl-based extractor Playwright shells out to, and Playwright only worked around it in 1.60.0 (microsoft/playwright#41000). tests/e2e/ui pins @playwright/test 1.58.1, so the pairing is unconditionally broken. The Node bump came from this fork, not upstream: a1b68cf moved this job from node-version "20" to "24.16.0" on 2026-06-16, three days after the last green run, and upstream is still on "20". Moving to the 22 LTS line clears the regression without dragging a shared upstream test suite through a four-minor Playwright bump, and keeps us off an EOL Node. Reproduced and confirmed locally with @playwright/test 1.58.1, same machine and same archive, varying only the Node version: Node 26.5.0 hangs at `extracting archive` indefinitely, Node 22.23.1 completes the install in 16s. The timeout and retry guards from #26 stay in place so any recurrence fails fast and names the step instead of going silent.
blackflame007
merged commit Jul 28, 2026
538f336
into
litellm_internal_staging
72 of 73 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
test-server-root-pathhas been hard broken, not flaky. It last passed 2026-06-13 and was cancelled on all 28 runs since 2026-07-04, on both matrix legs, always at the same point: the Chrome-for-Testing zip reaches 100% andnpx playwright installnever returns, burning the full 30m job cap without ever asserting anythingHow it solves it:
The job is moved off Node 24.16.0 and onto the 22 LTS line. Node 24.16.0 regressed stream destruction (nodejs/node#63487), which deadlocks the yauzl-based extractor Playwright shells out to for the browser archive. Playwright only worked around it in 1.60.0 (microsoft/playwright#41000), and
tests/e2e/uipins@playwright/test1.58.1, so that pairing can never succeedRelevant issues
Upstream: microsoft/playwright#41000, nodejs/node#63487
Follows #26, which added the
timeoutandDEBUG=pw:installinstrumentation that made this diagnosableLinear ticket
Resolves NOL-88
Pre-Submission checklist
On tests: the regression test here is the job itself. It was silently non-executing for eight weeks and now runs its assertions again, which is the only signal that would have caught this
Screenshots / Proof of Fix
The
DEBUG=pw:installoutput added in #26 named the failing step precisely. Identical on all four retry attempts and both matrix legs of run 30401088569:The bytes arrive and the size matches
content-lengthexactly, so the transfer and the CDN object were never the problem; extraction isConfirmed locally by holding everything constant except the Node version, using the same
@playwright/test1.58.1 this repo pins:Node 26.5.0 stops dead and never returns; killed after 120s:
Node 22.23.1 completes the whole install in 16s:
The authoritative proof for this particular change is this PR's own
test-server-root-pathrun going green on both legs, since the job is the thing under repairType
Bug Fix
Infrastructure
Changes
One line of behaviour in
.github/workflows/test_server_root_path.yml:node-versionmoves from"24.16.0"to"22", with a comment recording why it must not go back to 24 or 26 while@playwright/testis below 1.60.0Worth noting where the breakage came from, because it argues for this fix over the alternative. The Node bump was ours, not upstream's: a1b68cf moved this job from
node-version: "20"to"24.16.0"on 2026-06-16, three days after the last green run, as a drive-by inside an unrelated fal_ai change. Upstream is still on"20". The other available fix is bumping@playwright/testfrom 1.58.1 to 1.60.0 or later, but that is a shared upstream test suite carrying the full Admin UI e2e specs, a four-minor bump would churn a lockfile this fork has to keep merging against, and it would not be validated by anything in this PR. Correcting our own regression is the smaller and better attributed changeThe
timeout, per-step caps andDEBUG=pw:installfrom #26 all stay. They are what turned a 30m silent cancellation into a labelled 7m failure, and they should keep any recurrence legibleQA runbook
Confirm both
test-server-root-path (/api/v1)andtest-server-root-path (/llmproxy)pass on this PR, and that theRun SERVER_ROOT_PATH redirect e2estep actually executes rather than being skipped, which is the step that had not run since JuneFinal Attestation