ci(codspeed): pin benchmark runner to ubuntu-24.04 - #31746
Conversation
ubuntu-latest resolves to different runner images between the BASE (main/staging) and HEAD (PR) runs, so CodSpeed reports 'Different runtime environments detected' and emits false-positive regressions (e.g. a -25.2% swing on test_completion_multi_turn in #31684, an MCP auth fix with no LLM code changes). Pinning the runner to a fixed image keeps BASE and HEAD on the same hardware so 1 ms swings on a ~3 ms benchmark stop blocking unrelated PRs. Fixes #31738
…itellm_/busy-lehmann-caafae
Greptile SummaryThis PR pins the CodSpeed benchmark runner from
Confidence Score: 5/5This is a one-line CI configuration change that only affects the benchmark runner image; no production code, tests, or schemas are touched. The change is minimal and self-contained — it replaces a floating runner label with a fixed Ubuntu image. The author verified the workflow completes successfully on No files require special attention.
|
| Filename | Overview |
|---|---|
| .github/workflows/codspeed.yml | Single-line runner pin from ubuntu-latest to ubuntu-24.04 to eliminate false-positive CodSpeed benchmark regressions caused by differing runner images between BASE and HEAD runs. |
Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile
Merging this PR will not alter performance
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The CodSpeed check flip-flops on internal staging and on PRs targeting it (e.g. "+11.75% improvement" on one run, "-25.36% regression" on the next) because the comparison flags "different runtime environments" and the benchmarks are only 3-4 ms, so sub-millisecond runner noise swings the result by 25-30%. Pinning the runner to ubuntu-24.04 in this PR helps the head side, but the internal_staging base is still recorded on the old unpinned runner, so comparisons keep flapping until the pin merges and the base is re-baselined. Until that settles, the red X's on internal staging make the OSS project look unhealthy and confuse contributors, so drop the litellm_internal_staging push and pull_request triggers and keep CodSpeed running on main only.
* ci(codspeed): pin benchmark runner to ubuntu-24.04 ubuntu-latest resolves to different runner images between the BASE (main/staging) and HEAD (PR) runs, so CodSpeed reports 'Different runtime environments detected' and emits false-positive regressions (e.g. a -25.2% swing on test_completion_multi_turn in BerriAI#31684, an MCP auth fix with no LLM code changes). Pinning the runner to a fixed image keeps BASE and HEAD on the same hardware so 1 ms swings on a ~3 ms benchmark stop blocking unrelated PRs. Fixes BerriAI#31738 * ci(codspeed): stop running benchmarks on litellm_internal_staging The CodSpeed check flip-flops on internal staging and on PRs targeting it (e.g. "+11.75% improvement" on one run, "-25.36% regression" on the next) because the comparison flags "different runtime environments" and the benchmarks are only 3-4 ms, so sub-millisecond runner noise swings the result by 25-30%. Pinning the runner to ubuntu-24.04 in this PR helps the head side, but the internal_staging base is still recorded on the old unpinned runner, so comparisons keep flapping until the pin merges and the base is re-baselined. Until that settles, the red X's on internal staging make the OSS project look unhealthy and confuse contributors, so drop the litellm_internal_staging push and pull_request triggers and keep CodSpeed running on main only. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
3 participants
Relevant issues
Fixes #31738
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
The CodSpeed workflow runs only on
push/pull_requesttomainandlitellm_internal_stagingplusworkflow_dispatch, so there is no live-proxy curl to show here; the realistic proof is a green run of the workflow itself on the pinned runner. I dispatchedcodspeed.ymlon this branch and it completed successfully onubuntu-24.04:https://github.com/BerriAI/litellm/actions/runs/28469560495
Type
🚄 Infrastructure
Changes
ubuntu-latestresolves to different runner images between the BASE (main/litellm_internal_staging) and HEAD (PR) runs, so CodSpeed reports "Different runtime environments detected" and emits false-positive regressions. A concrete example is the -25.2% swing reported ontest_completion_multi_turn(3.1 ms -> 4.2 ms) in #31684, an MCP auth-resilience fix that touched no LLM code pathPinning the benchmark job to a fixed image keeps BASE and HEAD on the same hardware, so sub-millisecond swings on a ~3 ms benchmark stop blocking unrelated PRs. This is the Phase 1 quick win from #31738
Making CodSpeed non-blocking / informational-only is a branch-protection setting in the GitHub repo rather than a code change, so it is intentionally left out of this PR