feat(ci): add the cost map guard check

[mateo-berri]

TLDR

Problem this solves:

• Cost map PRs only get a JSON parse and a schema-sync check today
• The backup copy of the map can drift from the root file unnoticed
• The upcoming cost map sync bot needs a check it cannot edit or bypass

How it solves it:

• New cost-map-guard check runs from the default branch on every PR
• It validates the root map, the backup copy, and the generated schema together
• On litellm_cost_map_sync_* branches it also enforces the bot contract
  • only the three map files change
  • no model or field disappears
  • sample_spec and fallback_generalizations stay untouched
• Retires test-model-map.yml, which the new check fully covers

User Flow

Before: a cost map PR that drops a model from the root file but not the backup passes every check and merges, so different proxies price the same model differently

1. A contributor opens a PR against https://github.com/BerriAI/litellm that removes openrouter/openai/gpt-3.5-turbo-instruct from model_prices_and_context_window.json and leaves litellm/model_prices_and_context_window_backup.json as it was
2. On https://github.com/BerriAI/litellm/pull/{n}/checks the validate-model-prices-json check is green: the file still parses and the schema is still in sync
3. A maintainer merges it
4. A proxy that pulls the map from GitHub on boot answers POST https://litellm-domain/v1/chat/completions for that model with no x-litellm-response-cost header, while a proxy running on the bundled backup still prices it from the old entry

After: the same PR is blocked with the exact drift named, and only merges once both copies agree

1. A contributor opens a PR against https://github.com/BerriAI/litellm that removes openrouter/openai/gpt-3.5-turbo-instruct from model_prices_and_context_window.json and leaves litellm/model_prices_and_context_window_backup.json as it was
2. On https://github.com/BerriAI/litellm/pull/{n}/checks the cost-map-guard check is red with litellm/model_prices_and_context_window_backup.json differs from model_prices_and_context_window.json; copy the root file over it
3. The contributor copies the root file over the backup and pushes; the check turns green
4. A maintainer merges it, and every proxy, whether it boots from GitHub or from the bundled backup, prices that model the same way

Design

This is the first of two PRs for the cost map sync bot. The bot (next PR) opens PRs on litellm_cost_map_sync_* branches from the live OpenRouter and Vercel AI Gateway catalogs and merges them once this check is green, so the check has to come from a branch the bot cannot rewrite. That is why it uses pull_request_target: GitHub runs the workflow and checks out the code from the default branch, and the PR's files are only ever read as data by ci_cd/cost_map_guard.py (base and head are fetched by SHA and read with git show, nothing from the PR is executed). The job keeps contents: read and does not persist credentials

Human PRs get the file checks only: the root map parses to an object, the backup is byte-identical to it, the checked-in schema equals the one ci_cd/generate_model_prices_schema.py would generate, and every entry validates against that schema. Bot branches additionally get the contract above, so a catalog glitch or a bad transform can add or reprice models but can never delete one, drop a field a customer relies on, touch code, or edit the special root keys

Relevant issues

Linear ticket

Part of LIT-6995 (the sync bot PR that follows resolves it)

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Three throwaway branches off this PR's tip play the PRs the check has to judge. Each was pushed and opened as a PR into this branch (#39830, #39831, #39832, closed again): pull_request_target only runs the workflow from the default branch, so the live check cannot fire on those and the runs below are the exact command the job runs, executed locally against the pushed revisions. The live run gets captured on the same branches once this lands on the default branch, before the bot PR merges

• litellm_cost_map_demo_human (acc5eaf): removes openrouter/openai/gpt-3.5-turbo-instruct from the root map only
• litellm_cost_map_sync_demo_ok (e5f43ba): a bot branch that adds openrouter/inception/mercury-2.5-preview and reprices openrouter/deepseek/deepseek-v4-pro-0813, with backup and schema in sync
• litellm_cost_map_sync_demo_bad (9bed767): a bot branch that removes a model, drops supports_reasoning from an entry, adds a key to sample_spec, and edits README.md

Before (b3c867c)

Human PR that de-syncs the backup

1. git worktree add /tmp/demo litellm_cost_map_demo_human && cd /tmp/demo && jq empty model_prices_and_context_window.json; echo "jq exit $?"
2. jq exit 0
3. python ci_cd/generate_model_prices_schema.py --check; echo "schema check exit $?"
4. model_prices_and_context_window.schema.json is in sync and model_prices_and_context_window.json validates against it then schema check exit 0: both steps of test-model-map.yml pass and the drift merges

Bot PR within the contract

1. Same two commands on litellm_cost_map_sync_demo_ok
2. Both pass (no bot contract exists yet, so a good and a bad bot PR are told apart by nothing)

Bot PR that breaks the contract

1. Same two commands on litellm_cost_map_sync_demo_bad
2. Both pass: the removed model, the dropped field, the sample_spec edit, and the README.md edit all go unnoticed

After (61bed79)

Human PR that de-syncs the backup

1. python ci_cd/cost_map_guard.py --base litellm_cost_map_guard --head litellm_cost_map_demo_human --head-ref litellm_cost_map_demo_human; echo "exit $?"
2. Output:

   cost map guard failed (human PR, file checks only):
   - litellm/model_prices_and_context_window_backup.json differs from model_prices_and_context_window.json; copy the root file over it
   exit 1
   

Bot PR within the contract

1. python ci_cd/cost_map_guard.py --base litellm_cost_map_guard --head litellm_cost_map_sync_demo_ok --head-ref litellm_cost_map_sync_demo_ok; echo "exit $?"
2. Output:

   cost map guard passed (bot contract enforced)
   exit 0
   

Bot PR that breaks the contract

1. python ci_cd/cost_map_guard.py --base litellm_cost_map_guard --head litellm_cost_map_sync_demo_bad --head-ref litellm_cost_map_sync_demo_bad; echo "exit $?"
2. Output:

   cost map guard failed (bot contract enforced):
   - bot PRs may only change the cost map files, not README.md
   - bot PRs may not remove models: openrouter/openai/gpt-3.5-turbo-instruct
   - bot PRs may not remove fields: openrouter/deepseek/deepseek-v4-pro-0813.supports_reasoning
   - bot PRs may not change sample_spec
   exit 1
   

Type

🚄 Infrastructure
✅ Test

Caveats (if any)

Medium

• The check first fires on PRs opened after this lands
  • pull_request_target reads the workflow from the default branch
  • so this PR itself shows no cost-map-guard run
• cost-map-guard replaces validate-model-prices-json wherever a ruleset or reviewer expects the old name
  • neither is a required check today

Low

• The three litellm_cost_map_demo* branches are throwaway, deleted after the live demo
• The declared_reasoning_efforts docstring in litellm/router_utils/reasoning_effort_capability.py still names validate-model-prices-json
  • feat(ci): add the cost map sync bot for openrouter and vercel_ai_gateway #39845 renames it to cost-map-guard
• A human PR that edits the root map without copying it over the backup now fails the check
  • the retired check let that drift through; the failure message names the fix
  • the last 15 map commits on staging all updated both files
• proxy-infra / Run tests (Python 3.10) is red at 61bed79, inherited from staging
  • test_http_parsing_utils.py::TestNumericFormFields::test_qualifiers_and_optionality_are_unwrapped fails the same way on every branch cut before fix(proxy): strip every TypedDict qualifier before numeric form-field detection #39780 (merged to staging 2026-09-05 01:36Z; staging's Unit Tests run 33956401452 is green since); the required context is the aggregate proxy-infra / Run tests, which is green, and the merge result carries the fix

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Uses pull_request_target (documented as base-branch-only execution with read-only PR data), which is security-sensitive CI surface area; changes gate merges on pricing map consistency and bot behavior.

Overview
Replaces the validate-model-prices-json workflow (test-model-map.yml, jq + schema --check) with a cost-map-guard job that runs on pull_request_target so the check always executes from the default branch while PR map files are read via git show at merge base and head.

The new ci_cd/cost_map_guard.py enforces, on every PR: valid root model_prices_and_context_window.json, byte-identical litellm/model_prices_and_context_window_backup.json, checked-in schema matching generate_model_prices_schema.py, and schema validation of entries. Branches named litellm_cost_map_sync_* additionally get a bot contract: only the three map files may change, no model/field removals, and sample_spec / fallback_generalizations must be unchanged.

Adds tests/test_litellm/test_cost_map_guard.py covering file drift, validation errors, bot vs human rules, and CLI/git integration.

^{Reviewed by Cursor Bugbot for commit 61bed79. Bugbot is set up for automated code reviews on this repo. Configure here.}

• 61bed79 passes /live-pr-risk

[greptile-apps]

Greptile Summary

This PR replaces the existing model-map validation workflow with a base-branch cost-map guard

• Verifies the root map, bundled backup, and generated schema remain synchronized
• Restricts sync-bot branches to the three map artifacts and prevents model, field, or special-root-key removal
• Adds focused unit and local Git integration coverage for human and bot pull requests
• Uses read-only pull_request_target execution without checking out or executing pull request code

Confidence Score: 5/5

The PR appears safe to merge, with no concrete correctness, security, or repository-rule violations identified

The workflow executes trusted base-branch code with read-only permissions, treats pull request revisions as data, and the guard and tests cover the stated synchronization and bot-contract requirements

Important Files Changed

Filename	Overview
.github/workflows/cost-map-guard.yml	Adds the read-only base-branch workflow that fetches pull request revisions and runs the guard
ci_cd/cost_map_guard.py	Implements synchronization checks and the stricter sync-bot change contract
tests/test_litellm/test_cost_map_guard.py	Covers file drift, schema validation, bot restrictions, and revision-based command execution
.github/workflows/test-model-map.yml	Removes the narrower validation workflow superseded by the new guard

_{Reviews (1): Last reviewed commit: "feat(ci): add the cost map guard check" | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 61bed79. Configure here.}

[tin-berri]

LGTM