Repository navigation
feat(decisions): backport #45474, #45811, #45734, #46594, and #46596 to rc/1.106.0 - #46628
Conversation
#45474) * refactor(decisions): rename the evaluation model mode to decisions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): canonicalize the health check mode without rebinding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): replace canonical_model_mode with is_decisions_model_mode Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): preserve evaluation mode compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle from this PR Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(dashboard): format decision mode helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Backport of #45474 to rc/1.106.0 Cherry-picked from 4c532a5 (main) Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): support Clef-omni audio and video * fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes The clip patterns now mirror Cloudflare's input schema, so a data URL or content_type carrying parameters like audio/webm;codecs=opus is accepted. The four proxy routes get media tests of their own, the backup cost map is synced with the new Clef-omni rows, and the docs page moves to the docs repo as a follow-up. * fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields * fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation * fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models * fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers * test(decisions): cover the Fireworks System One router in the media refusal cells --------- Backport of #45811 to rc/1.106.0 Cherry-picked from a86ad9e (main) Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…ield (#45734) * feat(decisions): send images to Cloudflare Clef OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise. * fix(decisions): follow drop_params for images and use the call timeout for image downloads * fix(decisions): drop a malformed images entry on its own under drop_params * fix(decisions): refuse images next to input whatever drop_params says * fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs Refuse more than 4 images for Cloudflare clef before any download, keep the caller's timeout as one deadline across image fetch retries, accept images next to input under drop_params, validate image strings and objects, and pass the caller's image URLs (not inlined bytes) to logging callbacks. * fix(decisions): check input_image URLs and clef image sizes * fix(decisions): accept data URL media type parameters and check clef image types * fix(decisions): count clef images after dropping unsupported types and flag clef vision --------- Backport of #45734 to rc/1.106.0 Cherry-picked from 2f3ccbe (main) Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): price typesafe/jev-1.14.0 TypeSafe released jev-1.14.0 on 2026-10-14 at the same price as jev-1.13.0 ($0.042 per 1M input tokens, output free; https://docs.typesafe.ai/models). Without an entry, calls through the TypeSafe pass-through log a cost of 0: TypeSafePassthroughLoggingHandler prices from litellm.model_cost["typesafe/<model>"], and a deployment's model_info price is registered under its model id only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cost): add max_input_tokens to the typesafe jev rows --------- Backport of #46594 to rc/1.106.0 Cherry-picked from 9d72bae (main) Co-authored-by: Mike Hyams <241361536+hyamie@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ext headers (#46596) * feat(typesafe): support Jev 1.14 images, box questions, and long-context headers * fix(cost): drop the typesafe/jev-1.14.0 entry main already prices * fix(typesafe): send images only with the images header and box questions only to models that answer them TypeSafe takes assets and input_image parts only when the request carries TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like any other provider that cannot take images. Box support now comes from a supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so jev-1.13.0 and jev-latest refuse or drop a box question before the call. A malformed provider_specific_header answers 400 instead of 500. * fix(typesafe): allow null box confidence and validate asset references and base64 * fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through --------- Backport of #46596 to rc/1.106.0 Cherry-picked from 3fec507 (main) Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Automatic reviews are disabled for this repository. |
|
|
|
@litereview code-review |
|
bugbot run |
✅ code-review5/5 · Ready for human reviewScope: full · Full PR review. Completed · Some checks not run. No correctness findings. The title needs a breaking-change marker. This backport includes #45734 ( Checks and execution details
Evidence:
Result: 1006 passed.
View review · Reviewed |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f3904e2. Configure here.
|
|
||
| def ir_to_openai_request(model: str, request: DecisionsIRRequest) -> Mapping[str, object] | UnsupportedDecisionsRequest: | ||
| if request.audio or request.videos: | ||
| return UnsupportedDecisionsRequest(reason="audio/video input is not supported by this Decisions provider") |
There was a problem hiding this comment.
OpenAI parser drops box answers
Low Severity
The OpenAI decisions response parser never matches a box question to a box answer, so a valid box result is turned into a refusal. Request conversion and outbound formatting already handle box questions, and the response adapter already accepts OpenAIBoxAnswer, but inbound _ir_answer has no box case. Any OpenAI-format provider that returns a box answer will look like it refused the question.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit f3904e2. Configure here.
There was a problem hiding this comment.
Real but pre-existing, this file is byte-identical to main after #46596. Fixing on main first so it rides the next rc, not this pick
3 participants
TLDR
Backport of #45474, #45811, #45734, #46594, and #46596 to
rc/1.106.0, so the next rc on this line ships the same change as mainProblem this solves:
rc/1.106.0is missing feat(decisions): accept decisions as a model mode alongside evaluation #45474feat(decisions): accept decisions as a model mode alongside evaluation (#45474)rc/1.106.0is missing feat(decisions): support Clef-omni audio and video #45811feat(decisions): support Clef-omni audio and video (#45811)rc/1.106.0is missing feat(decisions)!: send images to Cloudflare Clef through its images field #45734feat(decisions)!: send images to Cloudflare Clef through its images field (#45734)rc/1.106.0is missing fix(cost): price typesafe/jev-1.14.0 #46594fix(cost): price typesafe/jev-1.14.0 (#46594)rc/1.106.0is missing feat(typesafe): support Jev 1.14 images, box questions, and long-context headers #46596feat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596)How it solves it:
User Flow
#45474
Before: a deployment set to
model_info.mode: decisionsfails its health checktypesafe/jev-latestto config.yaml withmodel_info: {mode: decisions}Mode decisions not supportedAfter:
decisionsandevaluationbehave the sametypesafe/jev-latestwithmodel_info: {mode: decisions}model_info.modestill report"mode": "evaluation"in/model_group/info, same as today#45811
Before: a developer judging call recordings through Clef-omni gets answers that never heard the audio
"model": "clef-omni", astate, aquestionsmap, and"audio": ["data:audio/wav;base64,..."]usage.input_tokenscounts only the text, because the clip never reached Cloudflareinputplus aquestionslist) are dropped the same wayclef,clef-flash, an OpenAI decider) also come back 200 with the clips ignoredsupports_audio_inputorsupports_video_inputforclef-omniAfter: the same request judges the audio, and a model that cannot take clips says so
"audio": [...]and optionally"videos": [...], each clip a data URL or{"content_type": ..., "base64": ...}usage.input_tokensnow counts the media/systemoneand/decisionsroutes, reaches Cloudflare with the clips intactaudioandvideosand thedrop_paramshint, or, when the deployment or the proxy setsdrop_params: true, are dropped (malformed or not) and the text-only request goes out with a 200"audio": nullor"videos": nullis treated as no clips"supports_audio_input": true, "supports_video_input": trueonclef-omniand neither flag onclef-flash#45734
Before: a developer asking clef about a photo through the gateway gets a 400 on
/v1/decisions, and on/v1/systemonethe photo is silently left outcloudflare/clef(orcloudflare/clef-flash) model and gives the developer a keyinput_textpart, aninput_imagepart holding a PNG data URL, and a predicate questionimageslist next tostateandquestionsAfter: both requests reach clef with the photo, so its answers are about the image
cloudflare/clef(orcloudflare/clef-flash) model and gives the developer a keyinput_textpart, aninput_imagepart holding a PNG data URL, and a predicate questionprobabilityfor the predicate, and Cloudflare got the photo in itsimagesfieldimageslist, as a data URL, a{content_type, base64}object, or an https URL the gateway downloads firstimagesanddrop_params#46596
Before: a developer on Jev 1.14 cannot send an image, a box question, or long context through the proxy
assetsand the images headerboxquestion gets 400 "Input tag 'box' ... does not match any of the expected tags", on /v1/decisions toostatewith the long-context header still gets 400max_tokens_exceededAfter: the same requests reach Jev 1.14 with the image, the box question, and the header
assetsand the images header{"type": "box", "box": {"top": 0.11, "bottom": 0.52, "left": 0.49, "right": 0.82}, "confidence": 0.94}, and /v1/decisions with aninput_imagepart works toodrop_paramsBackport checklist
feat(decisions): accept decisions as a model mode alongside evaluation (#45474), cherry-picked from 4c532a5, same change as on mainfeat(decisions): support Clef-omni audio and video (#45811), cherry-picked from a86ad9e, adapted to the line intests/unit/decisions/test_main.pyfeat(decisions)!: send images to Cloudflare Clef through its images field (#45734), cherry-picked from 2f3ccbe, same change as on mainfix(cost): price typesafe/jev-1.14.0 (#46594), cherry-picked from 9d72bae, same change as on mainfeat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596), cherry-picked from 3fec507, adapted to the line inlitellm/proxy/_lazy_openapi_snapshot.json,tests/integration/_support/timing.py,tests/unit/decisions/test_main.pytests/unit/decisions/test_main.py,tests/unit/litellm_core_utils/test_health_check_helpers.py,tests/unit/litellm_core_utils/test_image_handling.py,tests/unit/llms/base_llm/decisions/test_base_decisions_transformation.py,tests/unit/llms/bedrock/chat/chat_completions/test_bedrock_chat_completions_transformation.py,tests/unit/llms/openai/decisions/test_openai_decisions_transformation.py,tests/unit/proxy/decisions_endpoints/test_endpoints.py,tests/unit/proxy/health_endpoints/test_health_endpoints.py,tests/unit/proxy/pass_through_endpoints/llm_provider_handlers/test_typesafe_passthrough_logging_handler.py,tests/unit/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py,tests/unit/proxy/test_litellm_pre_call_utils.py,tests/unit/systemone/test_main.py,tests/unit/test_model_prices_schema.py,tests/unit/test_typesafe_model_metadata.py,tests/unit/test_utils.py,tests/unit/types/test_types_utils.py,tests/unit/test_constants.py,tests/unit/llms/custom_httpx/test_llm_http_handler.py,tests/unit/test_main.py,tests/unit/proxy/pass_through_endpoints/test_llm_passthrough_endpoints.py, collected but not run here:tests/integration/management/test_model_health_check.py,tests/integration/providers/test_cloudflare_clef_images_chaos.py,tests/integration/providers/test_cloudflare_clef_images_wire.py,tests/integration/providers/test_cloudflare_clef_omni_media_chaos.py,tests/integration/providers/test_cloudflare_clef_omni_media_wire.py,tests/integration/providers/test_decisions_openai_format_wire.py, not run here:ui/litellm-dashboard/src/components/provider_info_helpers.test.tsx,ui/litellm-dashboard/src/lib/decisionModels.test.ts, the picks changelitellm-rust/, and this tool runs no cargo testsPre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
Screenshots / Proof of Fix
Live proof ran on main at the source tip of #46596 (ee9e86a), a 2-worker proxy on port 4046 with a real TypeSafe key from the Shared vault, before the merge, plus SDK legs at 3fec507
TypeSafe-Danger-Zone-Images: true, 512x512 PNG asset plus a box question answered 200 with a box answer (confidence 0.85)drop_paramsand 200 with a refusal slot underdrop_paramsTypeSafe-Danger-Zone-*input_imagepart (standard and URL-safe base64) and a box question answered 200 (predicate 0.97, box 0.85 and 0.89)On this line the picked unit tests (3166) pass against the line's own venv, and the e2e leg was skipped on Kerry's call since the change is self-contained
Type
🆕 New Feature
Caveats (if any)
Only on this line
Copied from #45474
Low
model_info.mode: decisionsand then runs on a proxy older than this PR still fails/healthwithMode decisions not supported. Only configs that opt in can hit thisdecisionsrows will returnMode decisions not supportedfrom/healthon proxies older than this PR. Real/v1/decisionsand/v1/systemonecalls to them still work, sincemodeonly picks the health probemode == "evaluation"exactly, so it needs its own change before it listsdecisionsmodels. Its bundled cost map snapshot also parses with a pinnedlitellm-model-catalog, so refreshing that snapshot afterdecisionsrows land needs the pin bumped past this PRevaluation, so a model added through the UI keeps working if the proxy is rolled backrole: "evaluation"is a different concept and is unchangedCopied from #45811
Medium
imageslist on a Decisions body passes validation and never reaches Clef-omni, because the request body allows unknown keys. Clef image support is feat(decisions): send images to Cloudflare Clef from /v1/decisions and /v1/systemone #45762, which touches the same files and needs a conflict resolution after this landsLow
audioandvideoson/v1/decisionsare a LiteLLM extension, not an OpenAI field;base64, is refused with the pattern message rather than a hint to use the object form, which takes parameterssupports_video_inputnow shows inget_model_infoandGET /model/infofor every cost-map entry that carries it, next tosupports_audio_input, which was already surfacedaudios,video) passes as an unknown key instead of a 400, the same as any unknown key on a Decisions body todaymodel_infoon a map without the Clef-omni rows has to use the canonicalcloudflare/@cf/cloudflare/clef-omniname, since the gate reads that key: on the shortcloudflare/clef-omnialias the flags show inGET /model/infobut clips still answer the 400input_imagetodayCopied from #45734
Medium
imagesto text-only decision models go from 200 to 400imagesnext toinputon any model, and router fallbacks to text-only deployments.drop_paramsbrings back the 200 with the images dropped, and the title carries!input_imagemessage no longer sits next to it on Cloudflareimagesfield has no positions, so a label such as "screen 2:" goes in the state and its image inimages. OpenAI keeps the parts in placeLow
input. Opt-in raw request logging and OTEL raw request spans record the wire body, the same as chat completions after it inlines an image=are dropped on downloadCopied from #46596
Low
assetsor box questions, a separate UI changeFinal Attestation
Note
High Risk
Large, behavior-changing Decisions surface (multimodal payloads, stricter 400s vs silent drops, remote image downloads, and opt-in danger-zone headers) on a critical proxy routing path.
Overview
Backports several main-line Decisions changes onto the rc line:
decisionsas a first-class model mode (withevaluationstill treated the same for health checks), plus a large expansion of what Decisions/System One requests can carry and how providers accept them.The Decisions call path now validates and transforms images (
input_image/images), assets, audio, and videos, with provider capability checks anddrop_paramsstripping (or 400 /UnsupportedParamsErrorwhen dropping would leave no input or questions). Box question/answer types are wired through IR, OpenAI, System One, proxy OpenAPI, and TypeSafe. Cloudflare Clef gets image limits, remote URL inlining, and animagespayload; Clef-omni gets audio/video on System One and OpenAI-shaped routes. TypeSafe Jev 1.14 usestypesafe-danger-zone-*headers (including proxy pass-through) to enable assets and box questions, converting OpenAI message images into referenced assets.Supporting updates include
supports_video_input/supports_box_questions, image URL fetch timeouts, separate logged vs wire bodies when images are inlined, model-catalogDecisionsmode, and cost-map entries for clef-omni and jev-1.14.0.Reviewed by Cursor Bugbot for commit f3904e2. Bugbot is set up for automated code reviews on this repo. Configure here.