Repository navigation
test: four tests stop asking litellm's live map about retired Groq models - #623
Merged
Merged
Conversation
…dels At 21:19Z on 2026-09-22 litellm removed models past their deprecation date from the model map on its GitHub main (BerriAI/litellm#42435). litellm fetches that file at import, so four tests that built a provider for groq/qwen/qwen3-32b or groq/llama-3.3-70b-versatile with no context window began raising UnknownContextWindow. No Zak Code change was involved: main's last CI ran at 19:40Z, before the upstream commit, and was green. Each test now declares the window its model had in litellm's bundled map (131000 and 128000), the idiom the neighbouring tests already use for self-hosted aliases. The Groq names stay: they are witnesses for "a metered cloud", and classification reads the provider prefix, not metadata. Measured: before, 4 failed / 37 passed in the two files against the live map; after, 41 passed against the live map and 41 passed with LITELLM_LOCAL_MODEL_COST_MAP=True; full suite 4555 passed, 0 failed. Not fixed here: pinning the bundled map for the whole suite fails 26 other tests (gpt-5.6, Gemini 3), so the suite's verdict still follows the live map. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
At 21:19Z today litellm removed "models past their deprecation date" from the model map on its GitHub main (BerriAI/litellm#42435). litellm fetches that file at import, so four tests that build a provider for
groq/qwen/qwen3-32borgroq/llama-3.3-70b-versatilewithout a context window now raiseUnknownContextWindow:test_local_only.py::test_metered_call_is_refused_at_the_request_buildertest_local_only.py::test_local_only_defaults_off_and_changes_nothingtest_local_only.py::test_extra_body_absent_by_defaulttest_prompt_cache_emit.py::test_groq_leaves_system_a_plain_stringNo Zak Code change was involved. Main's last CI ran at 19:40Z, before the upstream commit, and was green. From now on every CI run fails these four, on every branch.
The fix
Each test declares the window its model had in litellm's bundled map (131000 and 128000). The neighbouring tests already do this for self-hosted aliases. The Groq names stay, because they are only witnesses for "a metered cloud". Classification reads the provider prefix, not the model's metadata.
Evidence
LITELLM_LOCAL_MODEL_COST_MAP=True.Not fixed here: the suite still follows the live map
Pinning litellm's bundled map for the whole suite is not the fix. Under it, 26 other tests fail, because the gpt-5.6 and Gemini 3 tests need metadata only the live map has. So the suite's verdict depends on litellm's GitHub main in both directions. An offline run, or a GitHub outage, would read as 26 failures, because litellm falls back to that same bundled map when its fetch fails (measured here with the env var, not with the network down). A hermetic suite would pin the bundled map and register the few newer models the tests use from a small checked-in fixture. That is a larger change for its own PR.
🤖 Generated with Claude Code