chore: sync upstream 2026-08-21 - #208
Merged
shudonglin merged 297 commits intoAug 22, 2026
Merged
Conversation
…rsation system turns
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
Binding the bound method at import froze the module-level VertexBase instance, so callers that swap it no longer reached their replacement.
…/litellm into fix-anthropic-mid-conversation-system-cache
Taking the secret out of ~/.litellm/token.json stages a replacement and moves it into place, which needs room for a second file and a directory that will accept a new entry. A full disk refuses the first and a read-only ~/.litellm the second, and logout gave up there: it removed the file when it could, dropping the record that the keychain had never been confirmed clear, so the logout after it reported a clean keychain it never checked Shortening the file already in place needs neither, so the logout scrub and the legacy migration now fall back to overwriting it where it lies. On a read-only ~/.litellm the logout the user asked for now happens, instead of coming back with instructions to delete the file by hand
`lite whoami` led with "Authenticated" whenever a token file was on disk, even when the keychain holding the credential would not give it up. The notice about that sat below the account lines, so the session read as a working one and sent the user looking for the problem anywhere but the keychain
…arded auto_router marker params
An `auto_router/<alias>` marker entry's litellm_params (for example
`aws_region_name: eu-west-3`) were forwarded onto every routed call with
setdefault and then won the `{**litellm_params, **kwargs}` merge against
the selected tier's own values, so a Bedrock tier pinned to us-east-1 was
called in eu-west-3 and failed with 400.
The hook now records which keys it actually forwarded on the request's
metadata bucket, and `_update_kwargs_with_deployment` drops every
forwarded key the selected deployment defines itself, so marker params
only fill gaps a tier leaves open. Request-supplied values still win over
both. The stamp is stripped from logged metadata like its siblings.
Fixes BerriAI#37613
…t isolation
Five e2e tests over routes a customer drives through the gateway, each one
pinning a fix that currently has no live coverage.
The dedicated /openai_passthrough prefix used to be swallowed by the
provider-scoped /{provider}/v1/files and /{provider}/v1/batches routes, which
bound "openai_passthrough" as a provider name and failed inside the gateway
before ever reaching OpenAI. Two tests now upload a file and list batches
through that prefix and assert OpenAI's own objects come back.
Streamed /openai_passthrough/v1/responses and /openai_passthrough/v1/embeddings
are relayed to OpenAI but still have to be costed, since the customer budgets
against this traffic. Both used to land a row the gateway could not use: the
streamed responses call logged a zero-cost row under a random id, and
embeddings wrote no row at all. Each test now reconciles the logged spend and
token counts against the response the caller was actually served.
GET /v1/files narrowed its data to the caller's own rows but left first_id and
last_id addressing the shared provider account's page, handing any caller raw
provider file ids belonging to other tenants. The new test asserts both cursors
address rows in the page the caller can see.
ResourceManager.defer now accepts any callable rather than one returning None,
so a delete that answers with a response model can be deferred as-is.
…_6_cost_map feat: add bedrock grok 4.6 to model cost map
…ct credentials - accept the Live SDK's models/<id> and LiteLLM's vertex_ai/<id> when rewriting the setup model - keep a dict service account intact instead of stringifying it - treat same-target deployments holding different credentials as ambiguous - guard both websocket states before every close so a second close cannot raise - build the sendable close codes from the public CloseCode enum
…g flag The final streaming usage frame only carries usage.cost when the proxy runs with litellm_settings.include_cost_in_streaming_usage: true, and that flag is readable only off the module-level litellm setting. There is no header, key, or management route that turns it on per request, so a test cannot ask the shared e2e proxy for it, and the proxy's config does not live in this repo. The registry row stays as an uncovered gap with the reason recorded, rather than being deleted, so the behavior is still on the list of things we want covered once the gateway config is reachable. The StreamOptions model, ChatBody.stream_options, Usage.cost, and AnthropicMessagesResponse.id existed only for that test, so they go with it.
…d keep tier overrides and marker flags The forwarded-keys record moves off the shared metadata dict onto the per-request kwargs, where _update_kwargs_with_deployment consumes it, so sibling requests that share a metadata dict (abatch_completion) can no longer clear it mid-routing. Per-tier litellm_params from the hook response are never treated as forwarded marker params, and a deployment only beats a forwarded value when it sets its own, not when it carries a LiteLLM_Params default such as merge_reasoning_content_in_choices=False.
When a full disk refuses the replacement file and a read-only token file refuses the rewrite in place, the only way left to get the secret off disk is to remove the file carrying it. That file was also the note saying the keychain went unchecked, so its absence made the next logout read a keychain that was never confirmed as one already known to be clean. Removing it is what frees the room the replacement was refused for, so the note is written again on the way out and the logout after this one still warns.
…sation-system-cache fix: preserve prompt cache for mid-conversation system on unflagged Claude models
… extra `lite up` treats a token record whose key the keychain would not hand over as no login at all, and that clause had no test: every existing freshness test passed a record carrying a real key, so deleting the clause left the whole suite green The base install smoke check now also asserts keyring is absent, which is what makes the lazy import in cli_keyring meaningful. keyring ships in the cli extra only, so a plain `pip install litellm` must not be able to reach it
…essages bridge Both /v1/messages bridges (Responses API adapter for openai/* and the chat-completions adapter) now derive prompt_cache_key from the first 64 characters of metadata.user_id, next to the existing user mapping. The chat bridge only sets it when the resolved provider advertises prompt_cache_key in its supported params, so providers that reject unknown params are unaffected. A prompt_cache_key sent explicitly by the client always wins over the derived value. Fixes BerriAI#37508
The proxy's OAuth authorization server (dynamic registration, PKCE S256, loopback redirects, single-use codes, refresh rotation) gains a proxy-API audience: /authorize?resource=<proxy origin> renders a consent page with team selection and /token mints the same per-user credential lite login mints, so a native CLI can sign a user in through the system browser and call /v1/* with user and team attribution. Adds GET /.well-known/litellm-cli-auth as the versioned discovery contract for non-Python clients, POST /revoke (RFC 7009) for logout, and lite login --pkce, lite logout, and lite auth print-token on the CLI side. Proxy-API grants only ever redirect to a loopback address and the server never picks a team on the user's behalf. Fixes BerriAI#37332
The two tests that assert on reasoning cost read reasoning_tokens off the response and required it to be nonzero, without ever asking the model to reason. Both now send reasoning_effort, so the assertion rests on a parameter the test sets rather than on the model's default behavior. The cache-breakdown test sends it on its prime call too: OpenAI's prefix cache keys on the reasoning setting as well as the tokens, so priming at a different effort never produces a read.
A pre-flight that times out leaves its write parked inside the keychain, holding it against every later call, so the next read blocks on the main thread with no timeout of its own. Anything that resolves the credential more than once in a process hits it: an SDK Client built a second time never returns. The vault now remembers the silence and reports the keychain unreachable for the rest of the process rather than queueing behind the parked call.
…on GPT-5.6+ targets When the resolved deployment is provider openai and the model is GPT-5.6 or newer, the cache control hook now writes prompt_cache_breakpoint on the targeted content block and sets prompt_cache_options to explicit mode unless the caller already passed one. The /v1/messages bridges carry the marker through (the Responses bridge moves a marked system prompt into a developer message, since top-level instructions cannot hold one). Breakpoint counting and the stand-down check recognise both marker kinds, and client breakpoints already present in messages are no longer subtracted from the cap twice. Fixes BerriAI#37509
…bling records The discovery document is accepted only when its issuer is the --base-url the user typed and every endpoint and the resource share that origin (RFC 8414 section 3.3), so a tampered or redirected document can no longer point the code, verifier, or refresh token at another host. After a failed refresh the re-read token record is used only when it continues the same credential (same proxy, token endpoint, and resource) and has not expired, so a concurrent login against a different proxy can never hand this one its key
A login the keychain accepted whose token file could not be replaced left the superseded secret on disk, and the next load preferred the file unconditionally, so it served the old credential and erased the new one from the keychain on the way past. The keychain entry now carries the timestamp of the sign-in that minted it, and the two stores are compared on that instead.
The two stores hold different credentials on that path, so the rollback a migration does would hand the superseded one back out. The login that could not replace the file already named the state, and logout reports it too.
The stamp in the keychain entry is what decides that secret against one still sitting in the token file, and it came straight off the wall clock. A clock that stepped backwards between two logins therefore handed the win to the older of them: a login the keychain took but the token file could not be pointed at was resolved back to the credential it replaced, and the fresh one was erased from the keychain on the way past. save_cli_token now reads the stamp already on disk and pins the new sign-in just above it, so the ordering never depends on the clock having moved forwards. On a clock that did, this changes nothing.
The comment above the extra named cryptography as one of the heavy imports a thin install leaves out. That stopped being true when keyring joined the extra: on Linux it reaches the Secret Service through secretstorage, which depends on cryptography.
A login the keychain took but the token file could not record leaves the keychain naming a later sign-in than the file does. Reading only the file then stamps the next login below that keychain entry, and a clock that went back far enough puts the superseded credential back in use.
…ting a placeholder (BerriAI#37686) SCIM group members were matched against litellm user ids only. An identity provider that lists people by email or by the OIDC subject therefore matched nothing, and the member fell through to placeholder creation. Since BerriAI#37688 made a failed member creation fail the group sync rather than drop the member, that fallthrough is no longer quiet: the placeholder is created with user_email set to the member value, the duplicate-email check rejects it, and the whole group push answers 500. So on current staging a group listing anyone by their email fails outright, every other member in the payload included. An unmatched member id is now looked up across sso_user_id and user_email in one query. Searching either field first would hide a value that names one account by its SSO identity and another by its email, and hand the group to whichever was searched first. The two are not compared alike: an email is matched the way new_user matches one before accepting a new account, case-insensitively, because matching more strictly than the layer that would reject the placeholder is what turned an id whose casing differed from the stored email into that same 500. An SSO identity is matched exactly, since OIDC defines sub as case-sensitive and nothing folds its case on the way in. An exact user_id hit is checked the same way rather than trusted outright, since a value can be one account's id and another's SSO identity or email. That is not a corner case: the placeholders this bug provisioned are keyed by the very id the provider keeps pushing, so on a tenant that already has them the placeholder wins the id lookup and the real account can never be matched. Refusing names the problem instead of silently landing on the placeholder again. Those rows still have to be deleted before the real account resolves; making the sync heal itself needs a trustworthy way to tell a placeholder from an account someone created, and created_via lives in caller-writable metadata, so it is left to a follow-up. A value that names more than one account is refused with a 400 naming the id rather than attributed to one of them. Removals resolve too, since the roster holds canonical user ids and a directory removes people by the id it added them with. A removal counts the members one value names: the id as written when the roster holds it verbatim, which is how an earlier release recorded a member it could not match, together with the members it resolves to. Counting only the accounts on the roster keeps someone removable after a second account takes their email, which resolving table-wide would not, and counting both ways of naming a member together stops one value revoking two people when it is one member's canonical id and another's email. A value naming two of the group's own members is undecidable and fails rather than guessing or reporting a removal it did not perform. Resolves LIT-5383 Co-authored-by: Yassin Kortam <yassin@berri.ai>
…ream_cost_cache_tokens fix(cost): match streamed Messages usage cost to the recorded spend
…ons (BerriAI#37748) * test: enforce PT012 so a pytest.raises block cannot hide dead assertions `with pytest.raises(...)` stops at the first statement that raises. Anything sequenced after it inside the block never runs, so an assertion written there is never checked and the test still reports green. Two sites were doing exactly that, and both assertions turned out to be wrong once they started running. tests/llm_translation/test_prompt_factory.py asserted the bedrock rejection names "requires at least one non-system message", which holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup failure mentions "httpx.ConnectError", which never appears: the failure is an httpx.ConnectError whose message is "All connection attempts failed", so that test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since the old restore sat below the assertion and leaked the invalid URL into every later DB test the moment the assertion started being able to fail. The remaining 72 sites are rewritten without changing what they exercise: setup that cannot raise moves above the block, a nested `patch` moves outside it, and bodies with real control flow (a stream drain, an if/else on sync_mode, a retry loop) move into a local closure the block calls. Fixing PT012 unmasked two B017s, since ruff only reports a blind pytest.raises(Exception) once the block holds a single statement. tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException can_key_call_model actually raises. tests/local_testing/test_completion_cost.py was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true at some point; that dead first half is gone and the rest of the test, which checks medlm pricing resolves above zero, now runs instead of being skipped. * chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
…ning The cost map shipped cognition/swe-1.7 at $2.50 in / $12.50 out per million with $1.00 cache reads. Those are the Lightning numbers. Cognition's own model list at https://docs.devin.ai/desktop/models has uid swe-1-7 at $0.50 / $2.50 with $0.20 cache reads, and uid swe-1-7-lightning at $2.50 / $12.50 with $1.00 cache reads, so every swe-1.7 call has been costed at 5x since the entry landed. swe-1.7 now carries the standard rates and the Lightning tier gets its own entry, in both cost map copies. The source field on both moves to the desktop models page, which is the one that lists both tiers.
`members_with_roles` is a denormalized JSON snapshot written at add-time. `_update_team_members_list` backfilled `user_id` from `user_email` but never the reverse, so a member added by `user_id` alone was stored with `user_email=None` permanently - and `/team/info` returns that blob verbatim with no join to `LiteLLM_UserTable`, so the Admin UI's member table renders "-" for a user that plainly has an email. Fix both ends: - write path: `_resolve_member_identity` resolves identity both ways off the user rows the add just touched, so new roster entries stop being born blank. - read path: `/team/info` fills blank emails from `LiteLLM_UserTable` in one indexed `user_id IN (...)` query, repairing rows already in the database. Members that already carry an email are passed through untouched and cost no query, so this only ever turns a null into the right value.
Add sync and async regression tests for the BaseLLMHTTPHandler streaming path, which forwards provider response headers for the ~30 providers that ride the generic handler and had no coverage. Also drop redundant setup prose from the moonshot invoke test docstring.
…eam_spend_rows fix(streaming): price partial-stream spend rows at the real model and keep prompt and cache fields
…itellm_forward_bedrock_response_headers
…sponse_headers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…_keyed_pricing fix(fal_ai): price gpt-image-2 per size and quality from request params
…_model_name sagemaker_chat never put X-Amzn-SageMaker-Inference-Component on the request, so any endpoint backed by inference components answered 400 INFERENCE_COMPONENT_NAME_MISSING and the call never reached the container. The legacy sagemaker provider has built that header from model_id since BerriAI#8889, and this brings the chat provider in line. It goes on in validate_environment, which runs before the request is SigV4-signed, so the signature covers it The request body also always named the endpoint rather than the served model, which containers that validate the body's model answer with a 404. hf_model_name now becomes the body's model when it is set, and endpoints that do not set it keep sending exactly what they send today
…arametrize cases (BerriAI#37769) `pytest.raises(Exception)` with no `match=` passes on any error that broad. A TypeError from a refactor, a botched fixture, an import that moved: all of them read as the rejection the test claims to police, so the test goes green for the wrong reason and stays green after the behaviour it guards is gone. PT011 closes that gap for the 317 sites B017 could not reach, because B017 only fires on a single-statement body with no `as e` binding. Each pattern here is the message the code actually raised, recorded by running the sites under a plugin that logged the concrete type and text per call site, so the assertions describe observed behaviour rather than a guess. Where a site raises more than one message across its parametrize cases, the pattern is an alternation of what was seen; where the exception carries an empty `str()` and puts the text on `.message`, the site keeps a narrow `noqa` with the reason. PT014 removes four parametrize cases that were listed twice. The duplicate re-runs an assertion that already passed, and it usually marks a case someone meant to vary and forgot to edit.
…r_email fix: populate team member emails missing from the roster snapshot
…_response_headers fix(bedrock): forward provider response headers on chat completions
…we_1_7_pricing fix(cognition): price swe-1.7 at the standard tier, add swe-1.7-lightning
…inference_component_header fix(sagemaker_chat): send the inference component header and honor hf_model_name
# Conflicts: # .github/workflows/triage_rollout_heads_up.yml # tests/litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py # tests/old_proxy_tests/tests/bursty_load_test_completion.py # tests/old_proxy_tests/tests/load_test_embedding_100.py # tests/old_proxy_tests/tests/test_openai_request_with_traceparent.py
…ider wrapper) Pre-existing fork drift (present before this sync): the sync merge kept an orphaned QueryClientProvider import/mock/variable from a previous fork edit whose actual JSX usage upstream had already dropped when UsageTab stopped using react-query, leaving an unused-imports lint error. UsageTab.tsx has no react-query usage, so restoring the test file to match upstream exactly is correct.
Sync merge corruption (same class as the three prior -X theirs incidents documented in fork-patches.txt for this file): upstream independently restructured _apply_message_injections this cycle and the merge took upstream's simplified signature (no default_control parameter) while keeping the fork's inner-body reference to it, an F821 undefined-name. Restores default_control as an optional keyword param, threaded from the resolved per-request header default at the main pre_call_hook call site and the literal ephemeral default at the system-block call site, falling back to ChatCompletionCachedContent(type="ephemeral") when unset so the third (upstream-only) call site keeps its existing behavior unchanged.
…routes Pre-existing PT011 debt (file predates this sync unchanged), fixed while here since it was blocking the ruff-tests gate. Each match= pattern is verified against the exact ValueError message its call site's guard raises in litellm/proxy/pinned_provider_routes.py.
LIT001 22714->22942, LIT011 5585->5616: this sync's upstream changes to litellm/proxy/common_utils/reset_budget_job.py and litellm/proxy/spend_tracking/budget_reservation.py (byte-identical to upstream/litellm_internal_staging, verified via git diff) are the source; make lint-budget-update cannot reconcile a rule that grew, so the ceiling is bumped by hand to the CI-verified exact totals, same class as every prior ratchet bump in this file's history.
TQ005 2832->2836: the 5 newly flagged lines are byte-identical to upstream/litellm_internal_staging (verified via git diff), same class as the LIT001/LIT011 bump in the prior commit.
Sync merge corruption (same recurring class as this file's 2026-08-09/ 2026-08-14/2026-08-15 entries): upstream independently added its own second job-claim layer (_claim_job_for_costing / _release_job_claim, called from inside _track_completed_batch_cost right before the spend log write) this cycle, landing in disjoint hunks from the fork's pre-existing claim layer (_claim_job / _release_job_claim / _finalize_job, called earlier in check_batch_cost). The -X theirs merge auto-applied both with no conflict, stacking two claim layers on the same row: the outer claim always flips batch_processed=True first, so the inner claim's own compare-and-swap always found 0 rows and _track_completed_batch_cost always returned None without ever billing. Python also silently kept only the last of the two same-named _release_job_claim definitions, shadowing the fork's fenced one. Restores this file to byte-identical with the pre-sync commit for the affected region. A separate, genuine pre-existing gap in the fork's claim timing (unchanged since before this sync) is documented in fork-patches.txt and left for deliberate follow-up work rather than a rushed redesign under this fix.
Genuine upstream breakage from this sync's PR BerriAI#37692 (made the standalone entrypoint's prisma generate failure fatal by default instead of log-only), colliding with a real non-root permission constraint: prisma-python's generate() unconditionally re-copies schema.prisma into the installed package and chmod's the copy even when content already matches, and chmod requires owning the destination file, which no arbitrary runtime uid ever does for a file baked into the image at build time. Both runtime images already bake the generated client from this same schema.prisma at build time, so regenerating it at container start was always redundant work. Add LITELLM_PRISMA_CLIENT_PREBAKED to skip the runtime entrypoint's prisma generate call when set, and set it in both Dockerfile and docker/Dockerfile.non_root.
Tracks the prisma_migration.py/Dockerfile/Dockerfile.non_root fix from the previous commit so a future -X theirs sync doesn't silently drop it, consistent with this file's convention for confirmed upstream breakage fixed fork-side.
Sync merge artifact: two byte-identical FROM ... AS ui-builder blocks survived non-conflicting since Docker silently uses the later stage definition for a repeated name, so the build never broke, but it rebuilt the Admin UI twice per image build. Keep the single stage.
CheckBatchCost claimed a row before fetching its results, flipping batch_processed=True while the output file was still being read. That broke two things. The managed-files deletion guard only holds files whose batch still has batch_processed=False, so a concurrent delete could remove the very output file an in-flight costing run was reading. And a worker killed mid-fetch left the batch marked processed, so no other pod would select it until the abandoned-claim sweep ran. Move the claim to where upstream's own claim sits, inside _track_completed_batch_cost immediately before the spend-log write, without adopting upstream's simpler single-flag claim: the fork's status="pricing" fencing, reclaim sweep, spend-dedup marker and fenced finalize are all kept, so the crash-re-billing and disable_spend_logs protections are unchanged. Losing the compare-and-swap now costs only a duplicated results fetch, and is reported to the caller as CLAIM_LOST so it stays distinct from an unroutable row. Callers no longer release a claim across the fetch: a failed fetch never took one and leaves the row untouched, while a failed spend-log write releases before re-raising. This unblocks tests/proxy_unit_tests/test_check_batch_cost.py:: TestMultiPodBatchCostClaim, upstream's new class this sync cycle. Its journal and claim-call assertions are adapted to the fork's write shape while every behavioural assertion upstream makes is preserved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full -X theirs sync of BerriAI/litellm litellm_internal_staging (286 commits behind).
Merge conflicts
Resolved 4 modify/delete conflicts by taking upstream's deletion (test files upstream removed, none tracked in fork-patches.txt). Restored two fork-only files (
.github/workflows/triage_rollout_heads_up.yml,.github/scripts/triage_rollout_heads_up.py) that git's rename detection spuriously flagged as deleted during this sync's large upstream CI-workflow restructuring; upstream never had these files, so keeping the fork's version is the correct 3-way merge resolution.Fixes landed on top of the merge
UsageTab.test.tsxto upstream (dropped a deadQueryClientProviderwrapper left over from pre-sync fork drift)default_controlparameter threading inAnthropicCacheControlHook._apply_message_injections(sync merge dropped it, F821)pytest.raises(ValueError)calls intest_pinned_provider_routes.py(pre-existing PT011 debt, fixed while touching the ruff-tests gate)_claim_job_for_costing/_release_job_claimthis cycle, stacking on top of the fork's pre-existing claim layer with no merge conflict. The outer claim always won the compare-and-swap, so_track_completed_batch_costalways returnedNonewithout billing. Fixed by restoring this region to byte-identical with the pre-sync fork state.runtime-image/image-scanCI to fail: the standalone entrypoint's redundantprisma generateat container start alwaysPermissionErrors under a non-root uid. AddedLITELLM_PRISMA_CLIENT_PREBAKEDto skip it since both runtime images already bake the client at build time.ui-builderbuild stage inDockerfile(merge artifact; silently harmless since Docker uses the later stage, but rebuilt the Admin UI twice per image build)batch_processed=Truewhile the fetch was still in flight. The managed-files deletion guard only holds files whose batch still hasbatch_processed=False, so a concurrent delete could remove the very output file the costing run was reading, and a worker killed mid-fetch left the batch marked processed and unbillable by any other pod until the abandoned-claim sweep ran. This is a fork-design fix rather than merge-corruption cleanup, so it is written up separately below.Verified all documented fork-patches.txt entries with real functional content survived the merge (security pins, provider-pin hardening, path-injection guards, provider display-name UI patch, Dockerfile digest pins). Zero open code-scanning alerts, repo-wide and PR-scoped.
The claim-timing fix
Upstream added its own claim layer this cycle with deliberately different timing: claim after the fetch, immediately before billing. The duplicate-layer removal above deleted upstream's copy because two stacked claims silenced billing entirely, but that left the fork on its older claim-before-fetch timing, which
tests/proxy_unit_tests/test_check_batch_cost.py::TestMultiPodBatchCostClaim(upstream's new class) correctly fails.Rather than adopt upstream's simpler single-flag claim wholesale, this PR moves the fork's claim point and keeps the fork's hardening. The
status="pricing"fencing, the_reclaim_abandoned_pricing_claimssweep, the spend-dedup marker and the fenced finalize all stay, so crash-re-billing anddisable_spend_logsprotection are unchanged. Losing the compare-and-swap now costs only a duplicated results fetch, and is reported to the caller as aCLAIM_LOSTsentinel so it stays distinct from an unroutable row. Callers no longer release a claim across the fetch: a failed fetch never took one and leaves the row untouched outright, while a failed spend-log write releases before re-raising.The 5 previously failing tests now pass with every behavioural assertion upstream makes left intact. Only their journal and claim-call assertions were adapted to the fork's write shape, since the fork's claim carries
status="pricing"and its marker and fenced finalize are separate writes. One fork test (test_cost_tracking_failure_leaves_job_unprocessed, LIT-4008) was tightened from "the only extra write is the release" to "no row writes at all", the stronger form of the same guarantee under the new ordering.Verified by mutation: ignoring a lost claim fails
test_a_pod_that_loses_the_claim_after_fetching_does_not_bill, and dropping the release-on-billing-failure failstest_a_failed_spend_log_write_releases_the_claim. The ruff-strict, type-discipline, basedpyright and test-quality budget gates all pass unchanged, with no new ceilings needed