ci: triage chronic fork CI reds (zizmor, OSV, image scan, root-path tests) - #26
Merged
blackflame007 merged 1 commit intoJul 28, 2026
Merged
Conversation
…ests) Four checks had been failing on essentially every push or schedule against litellm_internal_staging, so a genuine regression could no longer be told apart from the standing noise. zizmor exited 13 on two action pins whose version comment did not match the commit they pinned: both used the moving `v6` tag as a comment while pinning setup-node v6.4.0's SHA. Repinned to v6.5.0 with an exact comment. osv-scan reported gitpython 3.1.52 (5 advisories, up to 8.8) plus postcss 8.5.13 and brace-expansion 5.0.7 on the dashboard. All three have fixed releases, so they are bumped rather than ignored. gitpython arrives through the optional mlflow extra and is not in the shipped image; postcss and brace-expansion are build/dev only, but a fix exists so there is no reason to carry them. Grype's image-scan findings (pypdf, pyasn1) were already resolved by the upstream sync in #24; only the daily schedule, which ran against the pre-merge commit, was still red. test-server-root-path was not asserting anything. Since 2026-07-04 every run hung in `playwright install` immediately after the Chrome-for-Testing zip reached 100%, then died at the 30 minute job cap as "cancelled". The retry wrapper added in BerriAI#32406 cannot recover from that because no attempt ever returns. Each browser-install attempt is now bounded by `timeout`, the apt half is split out so a SIGTERM cannot strand dpkg, and every long step got its own cap so a failure names the step instead of surfacing as a bare cancelled job. DEBUG=pw:install is set so the next occurrence says where it stopped. The npm cache key also pointed at the dashboard lockfile while `npm ci` runs in tests/e2e/ui, so it never hit; it now points at the lockfile actually installed.
blackflame007
merged commit Jul 28, 2026
edf1586
into
litellm_internal_staging
73 of 77 checks passed
5 tasks
blackflame007
added a commit
that referenced
this pull request
Jul 28, 2026
… install hang (#28) `test-server-root-path` last passed 2026-06-13 and was cancelled on every run since 2026-07-04, on both matrix legs, always at the same point: the Chrome-for-Testing zip reaches 100% and `npx playwright install` never returns, burning the full 30m job cap. With DEBUG=pw:install (added in #26) the first post-mitigation run named the step exactly, identically on all four attempts and both legs: pw:install -- download complete, size: 175440843 pw:install SUCCESS downloading Chrome for Testing 145.0.7632.6 pw:install removing existing browser directory if any pw:install extracting archive <no further output; killed by `timeout 75`> So the transfer was never the problem; extraction was. Node 24.16.0 regressed stream destruction (nodejs/node#63487), which deadlocks the yauzl-based extractor Playwright shells out to, and Playwright only worked around it in 1.60.0 (microsoft/playwright#41000). tests/e2e/ui pins @playwright/test 1.58.1, so the pairing is unconditionally broken. The Node bump came from this fork, not upstream: a1b68cf moved this job from node-version "20" to "24.16.0" on 2026-06-16, three days after the last green run, and upstream is still on "20". Moving to the 22 LTS line clears the regression without dragging a shared upstream test suite through a four-minor Playwright bump, and keeps us off an EOL Node. Reproduced and confirmed locally with @playwright/test 1.58.1, same machine and same archive, varying only the Node version: Node 26.5.0 hangs at `extracting archive` indefinitely, Node 22.23.1 completes the install in 16s. The timeout and retry guards from #26 stay in place so any recurrence fails fast and names the step instead of going silent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
litellm_internal_staging, so nobody could tell a new red from the standing red; the chore: sync upstream/main → litellm_internal_staging (782 commits, v1.93.0 → main tip) #24 sync already cost us signal because one real failure had to be separated from the noise by handtest-server-root-pathnever asserted anything at all; it hung and got cancelled at the 30 minute job cap on every run since 2026-07-04, burning about an hour of runner time per PR across the two matrix legsHow it solves it:
Relevant issues
Linear ticket
Resolves NOL-81
Pre-Submission checklist
There are no unit tests here because nothing in
litellm/changed; the checks themselves are the test, and the proof below is each scanner run to completion locally against this branchScreenshots / Proof of Fix
zizmor, same version and severity floor the workflow uses, run against this branch:
Before this branch it exited 13 on:
osv-scanner, same pinned version and same invocation as the workflow:
On
litellm_internal_stagingthe same command reports 7:Lockfile still installs cleanly after the surgical bump:
The Chrome-for-Testing object the root-path job stalls on is intact, which is what rules out a truncated download and points at the post-download handoff:
175440843 bytes is exactly the "167.3 MiB" the progress bar reaches before every run goes silent
Type
Infrastructure
Changes
zizmorwas a real finding, though a small one. Two workflows pinnedactions/setup-nodeto v6.4.0's SHA while commenting it# v6, and the movingv6tag has since advanced to v6.5.0, so the comment named a different commit than the pin. Every other pin in the tree comments the exact tag whose commit it pins, which is why only these two were flagged. Both are repinned to v6.5.0 with an exact comment, which also picks up that release's own dependency updatesosv-scanwas a real finding and is fixed rather than ignored, since every advisory had a published fix. gitpython goes to 3.1.56 throughuv lock --upgrade-package gitpython, which touched four lines and nothing else. It reaches us through the optional mlflow extra, so it is not in the shipped image, but there is no reason to carry five advisories for a lockfile bump. On the dashboard, postcss goes to 8.5.18 and the brace-expansion override to 5.0.8. Those two are hand-applied topackage-lock.jsonrather than regenerated, becausenpm install --package-lock-onlyon current npm also prunes three unrelated optional peer entries; that drift reproduces with zero package.json changes, so it does not belong in a security fiximage-scanwas a real finding that is already resolved. Grype flagged pypdf and pyasn1 in the runtime image, and the upstream sync in #24 moved both to fixed releases. Only the daily schedule stayed red because it ran against the pre-merge commit; the image-scan run on #24's head passed every step including the grype gate. Nothing was needed here beyond confirming ittest-server-root-pathwas a real regression that the chronic red had been hiding. It last passed on 2026-06-13 and has been cancelled on all 28 runs since 2026-07-04, both matrix legs, always at the same place:npx playwright installprints the Chrome-for-Testing progress bar up to 100 percent and then never returns, and the job dies at its 30 minute cap. The retry wrapper added in BerriAI#32406 cannot help, because a hang means no attempt ever returns for the loop to retry. So each browser-install attempt is now wrapped intimeout, which is what actually lets that retry loop do its job.install-depsis split out and left outsidetimeoutso a SIGTERM can never strand a half-applied dpkg state. The build, install and e2e steps each carry atimeout-minutessized to their real cost, which means the job-level cap should never be what fires, and a failure points at a step instead of showing up as an unexplained cancellation.DEBUG=pw:installis set so the next occurrence records where it stoppedOne unrelated bug fell out of reading that job: the npm cache was keyed on
ui/litellm-dashboard/package-lock.jsonwhilenpm ciruns intests/e2e/ui, so it could never hit. It now points at the lockfile that is actually installedI did not touch the
osv-scanner.tomlentry for diskcache. It is a correct waiver with a real reason and anignoreUntilof 2026-09-09 that forces a re-look, which is exactly the shape a waiver should haveQA runbook
GitHub Actions Security Analysisis green on this PR; it runs on every push, so it gates itselfOSV Scanis green on this PRUI Build Check,UI LintandUI Unit Testsare unchanged by the postcss bumpImage Scanonly runs on PRs that touch itspathslist, anduv.lockis on it, so it should run here and passtest-server-root-path, the honest expectation is that it now fails fast rather than passing. If the install still hangs, it should stop at 8 minutes with a named step timeout andpw:installoutput showing where, rather than a cancelled job at 30 minutes. If a retry does get through, the e2e assertion runs for the first time since June and its result is real signal either wayFinal Attestation