feat(decisions): accept decisions as a model mode alongside evaluation

[devin-ai-integration]

TLDR

Problem this solves:

• Decision models use mode: evaluation, which doesn't name the endpoint they serve
• Renaming the 16 existing cost map rows in place breaks released proxies, because they pull the cost map from main and their /health has no decisions handler
• So the plan is additive: existing rows stay evaluation, and new duplicate rows (follow-up PR) will use decisions. This PR makes the code ready for those rows

How it solves it:

• No cost map rows change. model_prices_and_context_window.json, its backup and cost_tracking_cases.json are identical to main
• One predicate in litellm/types/utils.py accepts both values:

  def is_decisions_model_mode(mode: str | None) -> bool:
      return mode in ("decisions", "evaluation")

• ahealth_check sends both values to the Decisions probe (handler_mode = "decisions" if is_decisions_model_mode(mode) else mode), and /health/test_connection accepts mode: decisions
• The dashboard isDecisionMode (Add Model provider lists, guardrail decision model picker) accepts both
• ModelInfoBase.mode lists both values
• The Rust litellm-model-catalog Mode enum gets Decisions. Catalog::parse rejects the whole file on an unknown mode, so without this one decisions row would make every Rust consumer of the cost map fail to load it
• The schema enum is untouched. ci_cd/generate_model_prices_schema.py builds it from the modes rows actually use, so it picks up decisions when the new rows land

User Flow

Before: a deployment set to model_info.mode: decisions fails its health check

1. The admin adds typesafe/jev-latest to config.yaml with model_info: {mode: decisions}
2. GET https://litellm-domain/health?model=that-model returns Mode decisions not supported

After: decisions and evaluation behave the same

1. The admin adds typesafe/jev-latest with model_info: {mode: decisions}
2. GET https://litellm-domain/health?model=that-model probes it with the Decisions call and reports it healthy
3. Deployments with no model_info.mode still report "mode": "evaluation" in /model_group/info, same as today

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy at 06c6cd6eb5, real TypeSafe API, no mocks, LITELLM_LOCAL_MODEL_COST_MAP=True, no DB

model_list:
  - model_name: typesafe-jev-no-mode
    litellm_params: {model: typesafe/jev-latest, api_key: os.environ/TYPESAFE_API_KEY}
  - model_name: typesafe-jev-decisions
    litellm_params: {model: typesafe/jev-latest, api_key: os.environ/TYPESAFE_API_KEY}
    model_info: {mode: decisions}
  - model_name: typesafe-jev-evaluation
    litellm_params: {model: typesafe/jev-latest, api_key: os.environ/TYPESAFE_API_KEY}
    model_info: {mode: evaluation}

curl -s -H 'Authorization: Bearer sk-45474-proof' http://127.0.0.1:4000/model_group/info
for model in typesafe-jev-no-mode typesafe-jev-decisions typesafe-jev-evaluation; do
  curl -s -G -H 'Authorization: Bearer sk-45474-proof' --data-urlencode "model=$model" http://127.0.0.1:4000/health
done

Model group	/model_group/info mode	/health
typesafe-jev-no-mode	evaluation	healthy_count 1, unhealthy_count 0
typesafe-jev-decisions	decisions	healthy_count 1, unhealthy_count 0
typesafe-jev-evaluation	evaluation	healthy_count 1, unhealthy_count 0

On main the typesafe-jev-decisions health check fails with Mode decisions not supported

Type

🆕 New Feature

Caveats (if any)

Low

• A config that sets model_info.mode: decisions and then runs on a proxy older than this PR still fails /health with Mode decisions not supported. Only configs that opt in can hit this
• The new decisions rows will return Mode decisions not supported from /health on proxies older than this PR. Real /v1/decisions and /v1/systemone calls to them still work, since mode only picks the health probe
• Lens (BerriAI/lens) matches mode == "evaluation" exactly, so it needs its own change before it lists decisions models. Its bundled cost map snapshot also parses with a pinned litellm-model-catalog, so refreshing that snapshot after decisions rows land needs the pin bumped past this PR
• The Add Model form still writes evaluation, so a model added through the UI keeps working if the proxy is rolled back
• The auto-router's dependency role: "evaluation" is a different concept and is unchanged

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

kerry seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 4/5

[Medium impact] Renames evaluation mode to decisions throughout the codebase.

The rename preserves legacy inputs, but required live-provider proof and the parameter-rebinding rule must be addressed before merging

Findings

1.  Live proof is missing ▶
2.  Parameter rebinding breaks repository rules ▶

Summary

Renames decision-model mode from evaluation to decisions across pricing data, health probes, Lens, and the dashboard

• Decision models use decisions while old evaluation configs still work.
• Lens finds decision models under the decisions mode.
• The model catalog labels decision models as decisions.

_{Reviews (1) · Last reviewed commit: "refactor(decisions): rename the evaluati..." · Reviewed by Greptile}

[greptile-apps]

Live proof is missing

Real-provider proof is still pending. Repository rules require live-proxy checks of inferred modes, legacy configs, and Lens selection. Add that proof before merging

Rule Used: Flag if the PR has no proof of e2e testing (if e2e testing is applicable here. If it's like an AGENTS.md change or something then probably no need to have e2e test proof) that either doesn't exist, claims that the box that it was trying to run in doe... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[devin-ai-integration]

Added live proof to the PR description: a real proxy against the TypeSafe API with no mocks, at merge base 1a42c5a and at c0bbbdb. It covers a cost-map decision model with no mode, an explicit mode: decisions deployment and a legacy mode: evaluation deployment, through /model_group/info, /health and a real /v1/systemone call

[greptile-apps]

Parameter rebinding breaks repository rules

This reassigns the mode parameter, which AGENTS.md forbids. Use a separate final value for the normalized mode before merging

Context Used: AGENTS.md (source)

[devin-ai-integration]

Fixed in cb75948: canonical_model_mode now wraps the existing mode or default_health_check_mode(...) expression, so the parameter is not rebound a second time

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_rename_evaluation_mode_to_decisions (46dfa61) with main (541e3ed)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[kerry-berri]

LGTM

[devin-ai-integration]

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

View session