fix(shadow_eval): split slot release from stops so stopped_at means exactly one thing

[tin-berri]

TLDR

Problem this solves:

• stopped_at does two jobs: index slot marker (sweeps) and stop marker (operators)
• Readers must guess which one a stamp means, and every guess has a race
• A stop racing a spend-budget crossing can read as natural completion

How it solves it:

• New internal released_at column takes over slot bookkeeping
• Sweeps write only released_at; any stopped_at now means a stop, always
• Spend and count arithmetic is only ever consulted for unstamped keys
• The migration moves old sweep stamps to released_at and re-anchors the index

User Flow

Before: whether a stamped eval reads "stopped" or "completed" depends on racy arithmetic

1. An admin stops a running eval from http://localhost:4000/ui/?page=cost-optimization just as its spend crosses max_budget
2. An in-flight sample lands moments before the stop is recorded, pushing recorded spend over the cap
3. The eval's card reads "completed", as if it ran its budget out; the admin's stop left no visible trace

After: a stamp is a stop, and no spend timing can reinterpret it

1. The admin stops the same eval at the same moment
2. The card reads "stopped" and keeps reading "stopped", whatever the recorded spend says
3. Evals that end by spending their budget still read "completed", before and after the slot sweep runs

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Setup: proxy from this repo against Postgres, config with auto_router1 (complexity router over gpt-5 family deployments) and a deterministic OpenAI-compatible upstream at :4299. Evals started with per-key max_budget (dollars); spend rows are topped up via SQL where noted because the stub's per-call cost is fractions of a cent

curl -sX POST http://localhost:4000/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" -d '{"key_alias": "spend-qa"}'
curl -sX POST http://localhost:4000/auto_router/shadow_eval/start -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"api_key_ids": ["<token_id>"], "router_name": "auto_router1", "shadow_percentage": 100.0, "judge_model": "gpt-5-nano", "duration_days": 7, "max_budget": 0.01}'
curl -s http://localhost:4000/auto_router/shadow_eval -H "Authorization: Bearer $LITELLM_MASTER_KEY"

Before (1d7f675)

A stamped job's label depends on spend arithmetic

1. Stop a running eval, then land two attempt rows timestamped one second before the stop, taking recorded spend to $16 against a $5 budget
2. GET /auto_router/shadow_eval: the frozen-count read includes the pre-stop rows, and on any leg whose stopped_by write raced or predated the column the derivation reads "completed"; the stamp's meaning is decided by arithmetic

The sweep writes the stop column

1. Let an eval spend past max_budget, then start a new eval on the same key
2. The sweep stamps the finished job's stopped_at: slot bookkeeping and stops share one column, which is what forces every reader to guess

After (71ff912)

Spend exhaustion reads completed with nothing stamped

1. Start with max_budget: 0.01, send 2 chats, top spend to $0.0202
2. GET list: status "completed", spend: 0.0202 of 0.01, stopped_at: null, stopped_by: null

The sweep frees the slot without touching the stop record

1. Start a second eval on the same key: 201; the first job's slot is released
2. GET list: first job still "completed", stopped_at still null

A stop outranks racing spend

1. Stop the running eval: 200, "stopped", stopped_by recorded
2. Insert 2 attempt rows dated one second before the stop, recorded spend $16 against the $5 budget
3. GET list: still "stopped"; spend is never consulted for a stamped key

A bare stamp from a pre-released_at pod reads stopped

1. Insert a leg shaped like an old pod's write mid-deploy: stopped_at set, no stopped_by, spend over budget
2. GET detail: "stopped"; a stamp means a stop, unconditionally

Type

🐛 Bug Fix

Caveats (if any)

• Adds one internal nullable column, released_at; not exposed in the API
• The migration moves actor-less sweep stamps off stopped_at, so their jobs keep reading completed
• Mixed-version residue: a job swept by a pre-released_at pod mid-deploy reads stopped until its window passes, the safe direction

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR separates shadow-evaluation slot release bookkeeping from explicit stop records.

• Adds released_at and moves the active-slot index, sweep, stop, and sampler queries to the new lifecycle marker
• Makes stopped_at take precedence over spend and attempt arithmetic when deriving status
• Migrates existing stamps and updates endpoint regression tests and generated schemas

Confidence Score: 4/5

The PR is not yet safe to merge because the data migration can erase legacy operator-stop intent and report those evaluations as completed

The new runtime split is internally consistent, but the migration clears every actor-less stopped_at value even though expired legacy operator stops were not guaranteed to receive stopped_by

Files Needing Attention: litellm-proxy-extras/litellm_proxy_extras/migrations/20260820150000_shadow_eval_released_at/migration.sql

Important Files Changed

Filename	Overview
litellm-proxy-extras/litellm_proxy_extras/migrations/20260820150000_shadow_eval_released_at/migration.sql	Adds and backfills released_at but unconditionally clears actor-less stop timestamps, including legacy explicit stops missed by the prior backfill
litellm/proxy/management_endpoints/auto_router_endpoints.py	Moves slot sweeps, active claims, and explicit stop release writes consistently to released_at
litellm/integrations/shadow_eval_logger.py	Restricts sampling to rows that are neither explicitly stopped nor internally released
litellm/types/management_endpoints/auto_router_endpoints.py	Gives stop stamps precedence over spend arithmetic while retaining window completion handling
tests/test_litellm/proxy/management_endpoints/test_auto_router_endpoints.py	Updates mocks and adds regression coverage for released sweeps and stamped-over-budget status, but does not exercise the legacy migration gap
schema.prisma	Adds the nullable internal released_at lifecycle field consistently with the two schema copies

_{Reviews (7): Last reviewed commit: "fix(shadow_eval): split slot release fro..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

bugbot run

[tin-berri]

@greptileai review pls

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a1df700. Configure here.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_shadow_eval_completed_status (71ff912) with litellm_internal_staging (e07a712)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (1d7f675) during the generation of this report, so e07a712 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e43e3bd. Configure here.}

[cursor]

Stop no longer frees the eval slot

Medium Severity

The active-job unique index and start lookup now key only on released_at, while sweeps only fill that column when the window or turn budget is already spent. Any stop that still writes stopped_at without released_at (rolling-deploy pods, or a stop that races the new SQL) leaves the key+direction slot occupied. Sampling already halted, but starting a replacement eval keeps returning 409 until ends_at, which can be days later.

Additional Locations (2)

• litellm/proxy/management_endpoints/auto_router_endpoints.py#L530-L538
• litellm/proxy/management_endpoints/auto_router_endpoints.py#L725-L731

 

^{Reviewed by Cursor Bugbot for commit e43e3bd. Configure here.}

[tin-berri]

@greptileai

[greptile-apps]

Legacy stop intent erased

If an expired legacy operator stop lacks stopped_by, this update clears its only stop marker, causing list and detail responses to report completion

Knowledge Base Used:

• Auto-router Shadow Evaluations
• Database Schema and Proxy DB Access Layer