Repository navigation
chore: backport CircleCI speedups (#43347) and cost-map fixes (#42951) to rc/1.104.0 - #43366
Conversation
|
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "input_cost_per_token_above_200k_tokens": 6e-06, | ||
| "output_cost_per_token_above_200k_tokens": 2.25e-05, | ||
| "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06, | ||
| "cache_creation_input_token_cost_above_200k_tokens_batches": 3.75e-06, |
There was a problem hiding this comment.
For a Claude Sonnet 4.5 batch request above 200k prompt tokens, this new cache-write rate applies, but the matching batch rates for ordinary input, output, and cache reads are missing from both Sonnet rows. The cost calculator keeps using their flat batch rates: at 250k prompt tokens, input is charged at 1.5e-6 per token rather than the 3e-6 rate implied by the above-200k price. This leaves batch spend undercounted. The matching tiers need schema and pricing-path support before these rates are published together.
Knowledge Base Used: Logging callbacks and telemetry
| litellm-rust/* | litellm/rust_bridge/* | rust-toolchain.toml | pyproject.toml | uv.lock | tests/windows_tests/* | .circleci/*) | ||
| has_windows_release=true ;; |
There was a problem hiding this comment.
The windows-release filter excludes ordinary litellm/*.py and packaged-data changes, so a PR changing only those files skips the full Windows wheel install-and-import check that previously ran on every PR. The remaining Linux check tests path lengths but does not install or import the wheel under a deep Windows prefix. This leaves Windows install and import failures unrelated to path length undetected for package-only changes.
* ci: cut CircleCI wall time without loosening test isolation * fix(ci): parse integration split files that follow --results The CircleCI machine image ships Python 3.12.2, whose argparse leaves the files positional empty when it follows an option and another positional, so every extensions node exited with 'unrecognized arguments'. Reproduced on 3.12.2; parse_intermixed_args selects the files on 3.12.2, 3.12.13 and 3.13 * test(ci): resolve command references in the Rust toolchain guard The Windows rustup install moved into the install_windows_toolchain command, which the guard only recognized for install_rust. It now accepts any command that installs a pinned rustup and reads the Windows toolchain pin from it * ci: cache the Windows release cargo build from main windows_release_wheel rebuilt every dependency with fat LTO on each run. It now restores the release target and cargo registry saved by main's scheduled run, drops the workspace crates' fingerprints so they always rebuild from the checked-out source, and still runs the full LTO link * ci: run the Windows release wheel build on windows.xlarge The fat-LTO release build is the slowest job in the pipeline; more cores speed up the dependency compile ahead of the final link * ci: skip the Windows fingerprint cleanup when the cargo cache missed On a cold cache the release fingerprint directory does not exist, and the CircleCI PowerShell wrapper failed the step on the suppressed not-found error (cherry picked from commit 635a718)
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| "input_cost_per_token_above_200k_tokens": 6e-06, | ||
| "output_cost_per_token_above_200k_tokens": 2.25e-05, | ||
| "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06, | ||
| "cache_creation_input_token_cost_above_200k_tokens_batches": 3.75e-06, |
There was a problem hiding this comment.
Medium: Long-context batch requests are underbilled
The calculator selects the input, output, cache-read, and cache-creation tiers independently, but this entry only defines the above-200K batch tier for cache creation. A user submitting an Anthropic batch above 200K prompt tokens is therefore charged the flat batch rates for the other components—for example, input remains 1.5e-06 instead of 3e-06. Add input_cost_per_token_above_200k_tokens_batches, output_cost_per_token_above_200k_tokens_batches, and cache_read_input_token_cost_above_200k_tokens_batches to this entry and the dated entry at line 14747.
PR overviewThis PR backports CircleCI performance improvements and model cost-map corrections to the rc/1.104.0 branch. One pricing gap remains open: Anthropic batch requests above 200K prompt tokens can be billed using lower flat rates for input, output, and cache reads. Users can trigger this directly through long-context batch requests, causing underbilling with a limited financial blast radius; no reported issues have yet been addressed. Open issues (1)
Fixed/addressed: 0 · PR risk: 5/10 |
f886a93 to
b5144fa
Compare
2 participants
TLDR
Backport of #42951 and #43347 to rc/1.104.0 with
git cherry-pick -x, so CircleCI on the release candidate can go greenProblem this solves:
integration-extensionsruns about 60 minutes on the rc and gets cancelledlocal_testing_part1fails the Bedrock cross-region parity check onanthropic.claude-mythos-previewHow it solves it:
integration-extensionsandunitacross 4 nodes and moves the Windows release wheel to its own jobAdaptation notes
#42951 is adapted. On the
claude-sonnet-4-5andclaude-sonnet-4-5-20250929rows it drops all four*_above_200k_tokens_batchespricing keys (input, output, cache read, cache creation), and the two matching parametrize cases intest_deployment_pricing_model_info_honors_a_200k_tier_batch_override. Three of those keys are declared by #42812 (native xAI batches and files), which is not on the rc, so keeping them fails the cost-map schema checks. The cache creation key is dropped too so the rows stay consistent: with only that one tier key, a batch request over 200k tokens was priced higher than on the rc. With all four gone, batch pricing for these models matches the rc tip exactly. The regenerated schema drops the now unused cache creation key. The rest of #42951 matches main; the pricing-key lists differ only in a neighbouring line from a PR the rc does not carryScreenshots / Proof of Fix
This PR only changes CI config and the cost map, so the proof is the affected suites run at the rc tip and at this branch with
LITELLM_LOCAL_MODEL_COST_MAP=True, plus this PR's own CircleCI runBefore (822ccfc)
pytest tests/unit/test_model_prices_schema.py tests/unit/test_utils.py tests/local_testing/test_get_model_info.py tests/unit/test_circleci_path_filter.py tests/unit/test_circleci_rust_toolchain.py tests/unit/test_pre_commit_lint.py tests/unit/router_strategy/test_router_tag_routing.py tests/windows_tests/test_check_windows_wheel_install.py tests/unit/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py tests/unit/litellm_core_utils/test_litellm_logging.py -n 831 failed, 1276 passed, includingtest_get_model_info_bedrock_cross_region_capability_parity, the same failure CircleCI showsAfter (b5144fa)
Same command
29 failed, 1297 passed, with no failure that is not also on the rc tip. The parity test now passespython ci_cd/generate_model_prices_schema.pyproduces no diff against the committed schemaA batch cost probe for
claude-sonnet-4-5(100k and 300k prompt tokens, with and without 50k cache creation tokens) returns the same cost here as on the rc tip: 0.15, 0.45, 0.16875 and 0.46875Behavioral review (gauntlet-mini, 5 lenses) survived after one fix: the cache creation tier key described above. The native-audio
max_input_tokenschange to 131072 is intentional upstream, per #42951Type
🚄 Infrastructure
🐛 Bug Fix