feat(ui): add the evaluation mode to the Add Model form by devin-ai-integration[bot] · Pull Request #45481 · BerriAI/litellm

devin-ai-integration · 2026-10-09T02:00:36Z

Summary

Resolves LIT-9365. Decision models (TypeSafe Jev and the others served on /v1/decisions and /v1/systemone) use model_info.mode: evaluation, but the Add Model form had no Evaluation option, and POST /health/test_connection rejected mode: "evaluation" with a 422. So the only way to add one was config.yaml

 TEST_MODES = [ ..., { value: "ocr", label: "OCR - /ocr" },
+  { value: "evaluation", label: "Evaluation - /v1/decisions" },
 ]

 test_model_connection(mode: Literal["chat", ..., "ocr",
+    "evaluation",
 ] | None)

No other backend change was needed. litellm.ahealth_check already maps evaluation to litellm.adecisions, so Test Connect probes it the same way /health does. schema.d.ts was regenerated with npm run gen:api

Tests

Unit tests cover test_model_connection passing mode="evaluation" through to ahealth_check, and the route returning 200 with a mocked TypeSafe /v1/systemone upstream instead of a 422. Dashboard tests cover the dropdown offering Evaluation - /v1/decisions and prepareModelAddRequest putting mode under model_info. A wire test in tests/integration/providers/test_decisions_wire.py checks that Test Connect with mode: "evaluation" reaches the TypeSafe decisions path. It is in the providers suite, which only runs in CircleCI's scheduled integration-providers job on main, so it has not run against this branch. It compiles locally but didn't run, because the integration rig env vars weren't set

Proof

Live proxy against the real TypeSafe API, LITELLM_LOCAL_MODEL_COST_MAP=True, request body shaped like the Add Model form ({"litellm_params":{"model":"typesafe/jev-latest","api_key":...},"mode":"evaluation"})

Step	main	this PR
`POST /health/test_connection`	422	200, `status: success`
`POST /v1/systemone` to the `jev` group	200	200
`POST /model/new` with `model_info.mode: evaluation`, then `GET /model/info`	n/a	200, stored `mode: evaluation`

E2E recording

Admin UI on a live proxy with Postgres and real TypeSafe at 20a1134: Add Model with TypeSafe, typesafe/jev-latest, Mode "Evaluation - /v1/decisions", Test Connect succeeds, save as jev-ui, Raw JSON shows model_info.mode: evaluation, Lens lists it as a System 1 model, and POST /v1/decisions returns 200. With Mode Chat, the same model fails Test Connect

Link to Devin session: https://app.devin.ai/sessions/102b3032e103426bb63709a63646ea10
Open in Devin Desktop: https://app.devin.ai/desktop/session/102b3032e103426bb63709a63646ea10?variant=devin
Requested by: @kerry-berri

Note

Devin errored when opening this Pull Request as kerry-berri.
As a fallback, Devin opened this PR as itself.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-09T02:00:40Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

CLAassistant · 2026-10-09T02:00:45Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

kerry seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2026-10-09T02:04:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-10-09T02:04:52Z

Confidence Score: 4/5

[Medium impact] Adds evaluation mode option to model testing form.

The evaluation flow looks sound, but the required Final annotations must be added before merging

Findings

Required `Final` annotations missing ▶

Summary

Adds evaluation to the Add Model dropdown and /health/test_connection, with matching generated API types

The Add Model form offers Evaluation mode for decision models.
Connection tests accept evaluation mode and probe decision models.

_{Reviews (1) · Last reviewed commit: "feat(ui): add the evaluation mode to the..." · Reviewed by Greptile}

greptile-apps · 2026-10-09T02:04:55Z

+    monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
+    litellm.in_memory_llm_clients_cache.flush_cache()
+
+    app = FastAPI()


Required Final annotations missing

The new test leaves app, client, upstream, and response without Final. The repository guide requires every variable to use Final; add these annotations before merging

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

codspeed · 2026-10-09T02:09:42Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_add_evaluation_mode_to_add_model_form (20a1134) with main (79c3de4)}

devin-ai-integration · 2026-10-09T02:22:37Z

UI e2e at 20a1134, run on a live proxy with Postgres and real TypeSafe: added typesafe/jev-latest as jev-ui with Mode "Evaluation - /v1/decisions". Test Connect succeeded, the saved model has model_info.mode: evaluation, Lens lists it as a System 1 model, and POST /v1/decisions returned 200. The same model with Mode Chat fails Test Connect, as expected

Evaluation Test Connect	Chat Test Connect

Saved mode	Lens

Written by Devin

…nd endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly

…45530) * test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone

feat(ui): add the evaluation mode to the Add Model form

20a1134

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Oct 9, 2026

View reviewed changes

kerry-berri approved these changes Oct 9, 2026

View reviewed changes

kerry-berri merged commit 0ae62c5 into main Oct 9, 2026
109 checks passed

kerry-berri deleted the litellm_add_evaluation_mode_to_add_model_form branch October 9, 2026 02:53

mateo-berri mentioned this pull request Oct 9, 2026

feat(decisions): add Laya and Bespoke providers to the Add Model form #45260

Merged

8 tasks done

yuneng-berri mentioned this pull request Oct 9, 2026

test: fix stale and flaky tests across CircleCI, GHA and Buildkite #45530

Merged

3 of 26 tasks

Navigation Menu

Repository navigation

feat(ui): add the evaluation mode to the Add Model form - #45481

feat(ui): add the evaluation mode to the Add Model form#45481
kerry-berri merged 1 commit into
mainfrom
litellm_add_evaluation_mode_to_add_model_form

devin-ai-integration Bot commented Oct 9, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Oct 9, 2026

Uh oh!

CLAassistant commented Oct 9, 2026

Uh oh!

codecov Bot commented Oct 9, 2026

Uh oh!

greptile-apps Bot commented Oct 9, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Oct 9, 2026

Uh oh!

codspeed Bot commented Oct 9, 2026

Uh oh!

devin-ai-integration Bot commented Oct 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

devin-ai-integration Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Proof

E2E recording

Uh oh!

devin-ai-integration Bot commented Oct 9, 2026

Uh oh!

CLAassistant commented Oct 9, 2026

Uh oh!

codecov Bot commented Oct 9, 2026

Codecov Report

Uh oh!

greptile-apps Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Oct 9, 2026

Merging this PR will not alter performance

Uh oh!

devin-ai-integration Bot commented Oct 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

devin-ai-integration Bot commented Oct 9, 2026 •

edited

Loading

greptile-apps Bot commented Oct 9, 2026 •

edited

Loading