Repository navigation
fix(fireworks_ai): restore supports_vision on minimax-m3 in the cost map - #41699
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 99b83a2. Configure here.
TLDR
Problem this solves:
fireworks_ai/minimax-m3tosupports_vision: falseHow it solves it:
supports_vision: trueon both Fireworks minimax-m3 cost map entriesUser Flow
Before: a developer sending an image to Fireworks minimax-m3 through the proxy gets a 400 saying the model cannot take images, while Fireworks itself answers the same request
fireworks-minimax-m3and one user message holding a text block plus animage_urlblocklitellm.BadRequestError: Fireworks AI model accounts/fireworks/models/minimax-m3 does not support image inputs. Use a Fireworks vision model or remove image_url content blocks.supports_vision: falseAfter: the same image request goes through the proxy and comes back with the image described
fireworks-minimax-m3and one user message holding a text block plus animage_urlblockusage.prompt_tokensincludes the image tokenssupports_vision: trueRelevant issues
Affected release
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Shared setup for every checkpoint below: a fresh worktree at that commit, its own venv, and a proxy booted with
--num_workers 2. Every image request goes to Fireworks for real and costs money. The four checkpoints bracket both the regression (the price sync merge) and this fix, so the flag flip is visible from both sidesmat570-proxy-config.yaml:.envholds onlyFIREWORKS_AI_API_KEY=<key>andLITELLM_LOCAL_MODEL_COST_MAP=True, so each proxy reads the cost map from its own checkout instead of fetching main's copy at bootBoot, from the worktree root:
set -a && source .env && set +a && python litellm/proxy/proxy_cli.py --config mat570-proxy-config.yaml --port <port> --num_workers 2chat.json:{"model":"fireworks-minimax-m3","max_tokens":400,"messages":[{"role":"user","content":[{"type":"text","text":"What animal is in this image? Answer with one word."},{"type":"image_url","image_url":{"url":"https://httpbin.org/image/png"}}]}]}messages.json:{"model":"fireworks-minimax-m3","max_tokens":400,"messages":[{"role":"user","content":[{"type":"text","text":"What animal is in this image? Answer with one word."},{"type":"image","source":{"type":"url","url":"https://httpbin.org/image/png"}}]}]}responses.json:{"model":"fireworks-minimax-m3","max_output_tokens":400,"input":[{"role":"user","content":[{"type":"input_text","text":"What animal is in this image? Answer with one word."},{"type":"input_image","image_url":"https://httpbin.org/image/png"}]}]}Checkpoint table (the Before and After blocks below are the merge base and the tip rows in full):
decbb96382supports_vision: truePig, prompt_tokens 155Pig, input_tokens 1Pig, input_tokens 155supports_vision: true87650bfe67supports_vision: false; responses never hit the gatedoes not support image inputsdoes not support image inputsPig, input_tokens 155supports_vision: falsee5aa10a1easupports_vision: false; responses never hit the gatedoes not support image inputsdoes not support image inputsPig, input_tokens 155supports_vision: false99b83a2d52supports_vision: truePig, prompt_tokens 155Pig, input_tokens 155Pig, input_tokens 155supports_vision: trueBefore (e5aa10a)
POST /v1/chat/completions with an image_url block
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:58535/v1/chat/completions -H "Authorization: Bearer sk-mat570-qa" -H "Content-Type: application/json" -d @chat.jsonOutput (long fields elided, everything else verbatim):
POST /v1/messages with an Anthropic image block
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:58535/v1/messages -H "Authorization: Bearer sk-mat570-qa" -H "Content-Type: application/json" -d @messages.jsonOutput (long fields elided, everything else verbatim):
POST /v1/responses with an input_image block
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:58535/v1/responses -H "Authorization: Bearer sk-mat570-qa" -H "Content-Type: application/json" -d @responses.jsonOutput (long fields elided, everything else verbatim):
GET /model/info reports supports_vision
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:58535/model/info -H "Authorization: Bearer sk-mat570-qa"Output (long fields elided, everything else verbatim):
After (99b83a2)
POST /v1/chat/completions with an image_url block
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:21664/v1/chat/completions -H "Authorization: Bearer sk-mat570-qa" -H "Content-Type: application/json" -d @chat.jsonOutput (long fields elided, everything else verbatim):
POST /v1/messages with an Anthropic image block
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:21664/v1/messages -H "Authorization: Bearer sk-mat570-qa" -H "Content-Type: application/json" -d @messages.jsonOutput (long fields elided, everything else verbatim):
POST /v1/responses with an input_image block
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:21664/v1/responses -H "Authorization: Bearer sk-mat570-qa" -H "Content-Type: application/json" -d @responses.jsonOutput (long fields elided, everything else verbatim):
GET /model/info reports supports_vision
curl -s -w '\nhttp=%{http_code}\n' http://127.0.0.1:21664/model/info -H "Authorization: Bearer sk-mat570-qa"Output (long fields elided, everything else verbatim):
Observations from the run, none introduced by this PR:
Type
🐛 Bug Fix
Caveats (if any)
Medium
supports_vision: falsefrom the litellm-providers storeLow
max_input_tokensvalues; out of scope hereLITELLM_LOCAL_MODEL_COST_MAPfetch main's map at boot and pick this up on mergeFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
99b83a2 passes /live-pr-risk