Repository navigation
chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job - #42608
Merged
Merged
Conversation
…r cron job The daily Claude Code compatibility-matrix job ran as a systemd timer on the litellm-compatibility-matrix-populator VM in the vertex-check GCP project. Replace that with a Render Docker cron job built from a new Dockerfile in tests/e2e/claude_code/cron_vm: pinned and checksummed debian base, gh, uv, and Claude Code CLI, a non-root populator user, and run_daily.sh as the entrypoint. run_daily.sh now clones a fresh blobless checkout per run (Render cron disks are ephemeral), reads the publish PAT from the github-token secret file under CREDENTIALS_DIRECTORY, and its comments no longer describe systemd. The .service and .timer units are gone; README.md and the env example describe the Render service, its secret files, and the local docker build instead.
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Render answers a bare 404 for the legacy pro_max plan name on a cron job (4c-16g is the same 4 CPU / 16 GB size) and the manual trigger route is /v1/cron-jobs, not /v1/cronjobs.
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
… tag from source The tag builds a Rust extension through maturin, which needs a C and Rust toolchain the cron image does not carry, so the first Render run failed at uv sync with "linker cc not found". Sync the locked dependencies with --no-install-project, install the PyPI wheel (what users run) with --no-build, and pass --no-sync to every uv run so uv never puts the source build back.
The validation run wrote the matrix into the image checkout, which nobody can read once the container exits. Save it under HOME and print it at the end of the log instead.
…0 docs PRs The docs repo has a few hundred open PRs, so a 100-item list never reached the week-old compat-matrix PR and the sweep left it open on every run.
Pushes never started a deploy during setup because Render only hears about them through its GitHub app, which the org does not have, so the README now carries the deploy command and the wait-for-live rule
The CI coverage gate requires every Dockerfile to be built by a job, and building this one on each PR that touches it also catches a broken pin or checksum before Render does
The five-file helper allowlist missed fixture_mode, which e2e_config now imports, so the first Render run died at conftest load with ModuleNotFoundError. Copy the image's whole tests/e2e tree instead and keep pytest from loading the EKS-harness conftest with --confcutdir
Contributor
Contributor
|
bugbot run |
…wn PRs The stale-PR sweep selected every open docs PR whose head branch starts with compat-matrix/, so a contributor's fork PR under that name would have been closed once a newer matrix PR existed. The sweep now resolves the publishing login from the token and only closes same-repo PRs that account opened
Contributor
Contributor
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 1f93010. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
run_daily.shcopies five helpers by name into the worktree, and main's helpers now importfixture_mode, so a fresh checkout of the script fails at conftest load (the VM ran a stale snapshot and never hit it)How it solves it:
gh,uv, and the Claude Code CLI are pinned and checksum-verifiedtests/e2etree and confines pytest's conftest loading toclaude_code/User Flow
Before: the populator lives on a GCE VM, so checking or rerunning it takes IAP SSH to that box
litellm-compatibility-matrix-populatorgcloud compute ssh litellm-compatibility-matrix-populator --zone us-central1-f --project vertex-check-481318 --tunnel-through-iapand read the last run withjournalctl -u litellm-compat-matrixsudo systemctl start litellm-compat-matrixon the VM and wait about 12 minutescompat-matrix/v1.101.0-2.1.228-2026-09-22and opens chore(compat-matrix): refresh for v1.101.0 + claude-code 2.1.228 litellm-docs#1622 with squash auto-merge on/etc/litellm-compat-matrix-github-tokenon the VM over SSHAfter: the populator is the Render cron job
litellm-compat-matrix, worked from the Render dashboard or its API0 6 * * *and the last run's status"status": "pending"compat-matrix/v1.102.0-2.1.228-2026-09-23and opens chore(compat-matrix): refresh for v1.102.0 + claude-code 2.1.228 litellm-docs#1643 with squash auto-merge ongithub-tokensecret file on the service's Environment tabtests/e2e/**they click Manual Deploy (or POST https://api.render.com/v1/services/crn-dapgapoae00c73d16130/deploys) before the next runRelevant issues
None
Affected release
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
Compat Matrix Imagejob builds the image and runs the pinned binaries on every PR that touches it; nothing here is unit-testable Python)uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more (no Python test files change; the image built locally with the samedocker buildthe job runs)@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Before (95d4613)
The populator is the VM's systemd timer, and its 06:00 UTC run today left docs PR #1266 open
gcloud compute ssh litellm-compatibility-matrix-populator --zone us-central1-f --project vertex-check-481318 --tunnel-through-iap --quiet --command 'systemctl list-timers litellm-compat-matrix.timer'... --command 'systemctl show litellm-compat-matrix -p Result,ExecMainStartTimestamp,ExecMainExitTimestamp'... --command 'journalctl -u litellm-compat-matrix --since "2026-09-22 06:00" -o short-iso | grep -E "==>|pull/"'The sweep closed nothing: with
--limit 100the list never reaches a two-week-old PR in a repo with about 360 open PRsgh pr list --repo BerriAI/litellm-docs --state open --limit 1000 --json number,headRefName,createdAt --jq '.[] | select(.headRefName | startswith("compat-matrix/")) | "\(.number)\t\(.headRefName)\t\(.createdAt)"'After, first pass (d244364)
The populator is the Render cron job
litellm-compat-matrix(crn-dapgapoae00c73d16130), deployed from this PR's tip and triggered by hand instead of waiting for 06:00 UTC.RENDER_API_KEYis a Render personal API key; the same calls work from the Render dashboard's Manual Run and Logs buttonsDeploy the tip and wait for it to go live
curl -s -X POST -H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" https://api.render.com/v1/services/crn-dapgapoae00c73d16130/deploys -d '{"clearCache":"do_not_clear"}', thencurl -s -H "Authorization: Bearer $RENDER_API_KEY" https://api.render.com/v1/services/crn-dapgapoae00c73d16130/deploys/dep-dapho2ijnfac73bpfj70 | jq '{id, commit: .commit.id, status, finishedAt}'Trigger a run
curl -s -X POST -H "Authorization: Bearer $RENDER_API_KEY" https://api.render.com/v1/cron-jobs/crn-dapgapoae00c73d16130/runs | jq '{id, status}'Read the run's step lines (same
==>markers as the VM journal above)curl -s -H "Authorization: Bearer $RENDER_API_KEY" "https://api.render.com/v1/logs?ownerId=tea-codhik0l5elc73foatl0&resource=crn-dapgapoae00c73d16130&startTime=2026-09-23T00:24:30Z&direction=forward&limit=100" | jq -r '.logs[] | "\(.timestamp[11:19]) \(.message)"' | grep -E '==>|pull/|passed'(paged onnextStartTimeuntil the run ends)The 80 passed and 17 skipped match the VM's last run on the same suite; the run resolved v1.102.0, which shipped after the VM's 06:00 run picked v1.101.0
Confirm how the run ended
curl -s -H "Authorization: Bearer $RENDER_API_KEY" "https://api.render.com/v1/services/crn-dapgapoae00c73d16130/events?limit=1" | jq '.[0].event'The docs PR it opened, and the stale one it closed
gh pr view 1643 --repo BerriAI/litellm-docs --json number,state,headRefName,autoMergeRequest,author --jq '{number, state, headRefName, autoMerge: (.autoMergeRequest != null), author: .author.login}'gh pr view 1266 --repo BerriAI/litellm-docs --json number,state,closedAt,headRefNamegh pr list --repo BerriAI/litellm-docs --state open --limit 1000 --json number,headRefName --jq '[.[] | select(.headRefName | startswith("compat-matrix/"))]'After (1f93010)
Same job, redeployed at the tip after the sweep was scoped to the publishing account's own PRs, and run again the same way
curl -s -H "Authorization: Bearer $RENDER_API_KEY" https://api.render.com/v1/services/crn-dapgapoae00c73d16130/deploys/dep-dapillp42hec73b2lqng | jq '{id, commit: .commit.id, status}'curl -s -X POST -H "Authorization: Bearer $RENDER_API_KEY" https://api.render.com/v1/cron-jobs/crn-dapgapoae00c73d16130/runs | jq '{id, status}'Step lines (same log call as above,
startTime=2026-09-23T01:27:28Z)The sweep found nothing to close: chore(compat-matrix): refresh for v1.102.0 + claude-code 2.1.228 litellm-docs#1643 from the run above had already auto-merged at 00:46Z, so this run's PR is the same-day refresh (
generated_atis the only cell that moved), and it auto-merged toogh pr view 1645 --repo BerriAI/litellm-docs --json number,state,mergedAt,author --jq '{number, state, mergedAt, author: .author.login}'The VM is gone and the project stays
gcloud compute ssh litellm-compatibility-matrix-populator --zone us-central1-f --project vertex-check-481318 --tunnel-through-iap --quiet --command 'sudo systemctl disable --now litellm-compat-matrix.timer && systemctl list-timers litellm-compat-matrix.timer --all'gcloud compute instances delete litellm-compatibility-matrix-populator --zone us-central1-f --project vertex-check-481318 --quiet && gcloud compute instances list --project vertex-check-481318 && gcloud projects describe vertex-check-481318 --format='value(projectId,lifecycleState)'Type
🚄 Infrastructure
Caveats (if any)
Medium
autoDeploy: yesnever firestests/e2e/**, deploy by hand (README shows the API call); the next scheduled run otherwise uses the old imageLow
gh2.101.0,uv0.10.9, Claude Code 2.1.228) bump through a PR with the new checksum--versionon the pinned binaries; it never runs the suite (real providers, about 12 minutes, real cost)--filter=blob:none) to keep it shortgh api user); the docs repo carries about 360 open PRs today--confcutdir=tests/e2e/claude_code, so the tree's top-levelconftest.pynever loads (the VM never loaded it either); thecoversmarker it registers stays unregistered, so collection prints onePytestUnknownMarkWarningper test, as beforemisc / Run testsjob was red at the tip ontests/test_litellm/test_utils.py::test_wrapper_async_does_not_fire_failure_hook_for_post_success_error, which fix(utils): isolate callback errors in async_post_call_success_deployment_hook #42535 broke onmainat 00:59Z (main's own Unit Tests runs failed on that test from https://github.com/BerriAI/litellm/actions/runs/35804440599 on); this PR changes only a shell script, a Dockerfile, a workflow, and docs. test(utils): raise the post-success hook error from a guardrail in the failure-hook regression #42646 fixed the test onmainat 02:11Z, and the PR reopened at the same tip is green on it: https://github.com/BerriAI/litellm/actions/runs/35810558292 (6563 passed, 25 skipped, on the merge into 5fa1257)/live-pr-risk
Verdict: no dependent path outside the cron job itself; the one dependent path (the container's daily run) was driven live on both sides
Breaking: none found
Backward incompatible: none for any proxy, SDK, dashboard, or config consumer; the diff touches no file that loads into a proxy process
Regression risk: a release whose PyPI wheel lags the GitHub tag fails at the install step (every release so far had the wheel first); a Claude Code CLI bump is a PR with a checksum instead of the VM's auto-updater
Dependency graph (repo-wide sweep for
run_daily.sh,cron_vm/,litellm-compat-matrix.*,CREDENTIALS_DIRECTORY,compatibility-matrix-populator)DockerfileCMD ->run_daily.sh: verified live (the Render run below)run_daily.sh->build_matrix.py,check_regressions.py(CLI output read back): verified live, same runrun_daily.shsweep ->gh pr list --author <login>on the docs repo: the exact query was run live as the publishing account (open PRs: none left; closed: [Feature]: Allow setting names for keys #1643 and [Feature]: Add new gpt-3.5-turbo-0125 [Week of Jan 28th] #1622 match, a fork PR under the prefix does not)tests/e2e/claude_code/passthrough/test_azure.py:12and_builder_unit_tests/test_matrix_builder.py:2name the env example andcron_vm/in docstrings only: still accurate.github/workflows/compat-matrix-image.yml->Dockerfile: verified on this PR's checkslitellm/,enterprise/,ui/, orlitellm/proxy/client/references any changed file, so no proxy A/B rig appliesNot verified: nothing skipped
1f93010 passes /live-pr-risk