Skip to content

ci: unpin Node 24.16.0 in the root-path e2e job to fix the Playwright install hang - #28

Merged
blackflame007 merged 1 commit into
litellm_internal_stagingfrom
litellm_nol88_playwright_install_hang
Jul 28, 2026
Merged

ci: unpin Node 24.16.0 in the root-path e2e job to fix the Playwright install hang#28
blackflame007 merged 1 commit into
litellm_internal_stagingfrom
litellm_nol88_playwright_install_hang

Conversation

@blackflame007

Copy link
Copy Markdown

TLDR

Problem this solves:

test-server-root-path has been hard broken, not flaky. It last passed 2026-06-13 and was cancelled on all 28 runs since 2026-07-04, on both matrix legs, always at the same point: the Chrome-for-Testing zip reaches 100% and npx playwright install never returns, burning the full 30m job cap without ever asserting anything

How it solves it:

The job is moved off Node 24.16.0 and onto the 22 LTS line. Node 24.16.0 regressed stream destruction (nodejs/node#63487), which deadlocks the yauzl-based extractor Playwright shells out to for the browser archive. Playwright only worked around it in 1.60.0 (microsoft/playwright#41000), and tests/e2e/ui pins @playwright/test 1.58.1, so that pairing can never succeed

Relevant issues

Upstream: microsoft/playwright#41000, nodejs/node#63487

Follows #26, which added the timeout and DEBUG=pw:install instrumentation that made this diagnosable

Linear ticket

Resolves NOL-88

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

On tests: the regression test here is the job itself. It was silently non-executing for eight weeks and now runs its assertions again, which is the only signal that would have caught this

Screenshots / Proof of Fix

The DEBUG=pw:install output added in #26 named the failing step precisely. Identical on all four retry attempts and both matrix legs of run 30401088569:

pw:install -- total bytes: 175440843
pw:install -- download complete, size: 175440843
pw:install SUCCESS downloading Chrome for Testing 145.0.7632.6 (playwright chromium v1208)
pw:install removing existing browser directory if any
pw:install extracting archive
<no further output; killed by `timeout 75`>

The bytes arrive and the size matches content-length exactly, so the transfer and the CDN object were never the problem; extraction is

Confirmed locally by holding everything constant except the Node version, using the same @playwright/test 1.58.1 this repo pins:

mkdir /tmp/pwrepro && cd /tmp/pwrepro
echo '{"name":"pwrepro","private":true,"devDependencies":{"@playwright/test":"1.58.1"}}' > package.json
npm install

# affected Node line
nvm use 26
PLAYWRIGHT_BROWSERS_PATH=./b26 DEBUG=pw:install ./node_modules/.bin/playwright install chromium

# LTS line this PR moves to
nvm use 22
PLAYWRIGHT_BROWSERS_PATH=./b22 DEBUG=pw:install ./node_modules/.bin/playwright install chromium

Node 26.5.0 stops dead and never returns; killed after 120s:

2026-07-28T22:30:14.274Z pw:install -- download complete, size: 170206961
2026-07-28T22:30:14.274Z pw:install SUCCESS downloading Chrome for Testing 145.0.7632.6 (playwright chromium v1208)
2026-07-28T22:30:14.275Z pw:install removing existing browser directory if any
2026-07-28T22:30:14.276Z pw:install extracting archive
STILL RUNNING after 120s -> HUNG

Node 22.23.1 completes the whole install in 16s:

2026-07-28T22:32:42.911Z pw:install SUCCESS installing Chrome Headless Shell 145.0.7632.6 (playwright chromium-headless-shell v1208)
2026-07-28T22:32:42.914Z pw:install validating host requirements for "chromium"
2026-07-28T22:32:42.914Z pw:install validation passed for chromium
2026-07-28T22:32:42.914Z pw:install validation passed for ffmpeg
2026-07-28T22:32:42.914Z pw:install validation passed for chromium-headless-shell
EXITED after ~16s

The authoritative proof for this particular change is this PR's own test-server-root-path run going green on both legs, since the job is the thing under repair

Type

Bug Fix

Infrastructure

Changes

One line of behaviour in .github/workflows/test_server_root_path.yml: node-version moves from "24.16.0" to "22", with a comment recording why it must not go back to 24 or 26 while @playwright/test is below 1.60.0

Worth noting where the breakage came from, because it argues for this fix over the alternative. The Node bump was ours, not upstream's: a1b68cf moved this job from node-version: "20" to "24.16.0" on 2026-06-16, three days after the last green run, as a drive-by inside an unrelated fal_ai change. Upstream is still on "20". The other available fix is bumping @playwright/test from 1.58.1 to 1.60.0 or later, but that is a shared upstream test suite carrying the full Admin UI e2e specs, a four-minor bump would churn a lockfile this fork has to keep merging against, and it would not be validated by anything in this PR. Correcting our own regression is the smaller and better attributed change

The timeout, per-step caps and DEBUG=pw:install from #26 all stay. They are what turned a 30m silent cancellation into a labelled 7m failure, and they should keep any recurrence legible

QA runbook

Confirm both test-server-root-path (/api/v1) and test-server-root-path (/llmproxy) pass on this PR, and that the Run SERVER_ROOT_PATH redirect e2e step actually executes rather than being skipped, which is the step that had not run since June

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

… install hang

`test-server-root-path` last passed 2026-06-13 and was cancelled on every run
since 2026-07-04, on both matrix legs, always at the same point: the
Chrome-for-Testing zip reaches 100% and `npx playwright install` never returns,
burning the full 30m job cap.

With DEBUG=pw:install (added in #26) the first post-mitigation run named the
step exactly, identically on all four attempts and both legs:

    pw:install -- download complete, size: 175440843
    pw:install SUCCESS downloading Chrome for Testing 145.0.7632.6
    pw:install removing existing browser directory if any
    pw:install extracting archive
    <no further output; killed by `timeout 75`>

So the transfer was never the problem; extraction was. Node 24.16.0 regressed
stream destruction (nodejs/node#63487), which deadlocks the yauzl-based
extractor Playwright shells out to, and Playwright only worked around it in
1.60.0 (microsoft/playwright#41000). tests/e2e/ui pins @playwright/test 1.58.1,
so the pairing is unconditionally broken.

The Node bump came from this fork, not upstream: a1b68cf moved this job from
node-version "20" to "24.16.0" on 2026-06-16, three days after the last green
run, and upstream is still on "20". Moving to the 22 LTS line clears the
regression without dragging a shared upstream test suite through a four-minor
Playwright bump, and keeps us off an EOL Node.

Reproduced and confirmed locally with @playwright/test 1.58.1, same machine and
same archive, varying only the Node version: Node 26.5.0 hangs at `extracting
archive` indefinitely, Node 22.23.1 completes the install in 16s.

The timeout and retry guards from #26 stay in place so any recurrence fails fast
and names the step instead of going silent.
@blackflame007
blackflame007 merged commit 538f336 into litellm_internal_staging Jul 28, 2026
72 of 73 checks passed
@blackflame007
blackflame007 deleted the litellm_nol88_playwright_install_hang branch July 28, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant