ci: fail on new unbounded SQL IN lists and add a Prisma chunking helper by ryan-crabbe-berri · Pull Request #42629 · BerriAI/litellm

ryan-crabbe-berri · 2026-09-23T00:46:14Z

TLDR

Problem this solves:

Postgres caps a statement at 32,767 bind parameters
An IN list built from table data binds one per value, so it breaks past the cap
That froze every due budget in the reset job (LIT-7535, [Bug]: End-user budget reset exceeds PostgreSQL's 32,767 bind-variable limit and never completes #40564)
Prisma only splits find_many, and gets take/skip/distinct wrong when it does
count, update_many and delete_many are never split
Nothing in CI stops a new unbounded list

How it solves it:

New litellm.repositories.chunked_in: find_many_in, count_in, update_many_in, delete_many_in
- Dedupes values, runs 5,000-value chunks in order, ANDs each with where
- Keyword-only chunk_size overrides the 5,000 default; outside 1 to 30,000 it raises ValueError before any query, leaving the rest of the filter headroom under the cap
- Works on a transaction handle; writes require an explicit atomicity choice
- Refuses a where that already filters the chunked field, and an update whose data writes it (a moved row would match a later chunk and update twice)
- Each chunk is sent as the same dict-and-list filter a hand-written call sends
check_unbounded_in_lists.py now fails CI on any finding not in the baseline
- 156 existing findings grandfathered in unbounded_in_baseline.txt
- Entries are keyed by file, function, field, the filtered expression and occurrence, not line numbers
- Swapping a baselined filter for a different expression on the same field reads as one new and one stale entry
- A stale entry also fails, so the baseline can only shrink
- --update-baseline regenerates it
Messages point at the helper for in and <> ALL($1::text[]) for not_in
A functional TypedDict("Name", {...}) field map is skipped: its "in" key is a field name, not a filter
One call site is migrated as proof of the API: _details_for_user_ids in key_metadata_recovery.py now uses find_many_in
- Up to 5,000 ids it sends the same single find_many with the same where; its existing tests pass unmodified

User Flow

Before: a contributor adds a membership filter that grows with a table and CI lets it merge

They write where={"user_id": {"in": user_ids}} in a proxy endpoint and open a PR
The Code Quality Checks job passes with no mention of the filter
On a large install the list passes 32,767 values and the request fails with too many bind variables in prepared statement

After: the same PR fails CI with the fix named, and the fix handles any list size

They write the same filter and open the PR
The Code Quality Checks job fails at check_unbounded_in_lists with path:line: prisma "in" filter over user_ids has no written bound ... Chunk it with litellm.repositories.chunked_in
They switch to find_many_in(table, "user_id", user_ids) (or record a real bound with # bounded-ok: <reason>) and the job passes
The same request with 40,000 ids now returns every row instead of failing

Relevant issues

Refs #40564

Linear ticket

Refs LIT-7535

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup: a local Postgres 14 on port 54917 with a probe schema holding a LiteLLM_Config table of 40,000 rows (id-0 to id-39999). demo.py sends a raw count(where={"param_name": {"in": ids}}) over all 40,000 ids, then calls each helper with the same ids (update_many_in inside db.tx())

Before (`7688f56`)

CI on a new unbounded list

python tests/code_coverage_tests/check_unbounded_in_lists.py
can't open file ... No such file or directory: nothing checks the filter, so the PR merges

40,000 ids through Prisma

DATABASE_URL=postgresql://postgres@127.0.0.1:54917/litellm_test python demo.py
Output:

raw count: DataError Assertion violation on the database: `too many bind variables in prepared statement, expected maximum of 32767, received 40001`
ModuleNotFoundError: No module named 'litellm.repositories.chunked_in'

After (`9c7eae6`)

CI on a new unbounded list

python tests/code_coverage_tests/check_unbounded_in_lists.py on the clean tree
Output: 156 unbounded IN list(s) (148 prisma, 8 raw-sql): 156 baselined, 0 new, 0 stale baseline entries. and echo $? prints 0
In a scratch copy of the tree, append async def scratch_violation(table, ids): return await table.find_many(where={"user_id": {"in": ids}}) to litellm/repositories/user_repository.py and rerun
Output, and echo $? prints 1:

litellm/repositories/user_repository.py:268: prisma `"in"` filter over `ids` has no written bound: it binds one parameter per value and Postgres caps a statement at 32,767. Chunk it with `litellm.repositories.chunked_in` (find_many_in / count_in / update_many_in / delete_many_in), or record the bound with `# bounded-ok: <reason>`

157 unbounded IN list(s) (149 prisma, 8 raw-sql): 156 baselined, 1 new, 0 stale baseline entries.

40,000 ids through Prisma

DATABASE_URL=postgresql://postgres@127.0.0.1:54917/litellm_test python demo.py
Output:

raw count: DataError Assertion violation on the database: `too many bind variables in prepared statement, expected maximum of 32767, received 40001`
count_in: 40000
find_many_in: 40000
update_many_in (in a tx): 40000
delete_many_in: 40000
rows left: 0

Tests and mutation check

pytest tests/unit/repositories/test_chunked_in.py prints 52 passed (an update writing the chunked field refused in any form, including Greptile's ['old', 'new'] / chunk_size=1 case, a chunk equal to a hand-written filter, sizes 0, 1, 5,000, 5,001 and 12,345, custom chunk_size setting the query count, chunk_size bounds 1 and 30,000 accepted and 0 / -1 / 30,001 refused before any query, dedup, sums, AND composition, same-field refusal, required atomicity, and a query-builder check that the composed filter renders like a hand-written one)
pytest tests/test_litellm/test_check_unbounded_in_lists.py prints 73 passed (new finding fails, stale entry fails, line shifts keep the key, a replaced expression on the same field fails, --update-baseline, helper exemption, constant spread, functional TypedDict field maps skipped while other calls' filters are still flagged)
- pytest tests/test_litellm/proxy/spend_tracking/test_key_metadata_recovery.py prints 34 passed: the 32 existing tests unmodified, plus 12,001 user ids going out as 5,000 / 5,000 / 2,001 chunks with every result merged, and a failing later chunk still logged and treated as no details
tests/integration/database/test_chunked_in_lists.py against real Postgres prints 5 passed: a raw 40,000-id count / update_many / delete_many is rejected while each helper handles the same ids, and attach_user_details fills every email for 40,000 users. It runs in the CircleCI database group
Mutation check per tests/AGENTS.md: 62 of 62 mutants killed (plus one equivalent mutant on the TypedDict skip), each restored to green afterwards, for example:
- chunk slice one short, or chunk size ignored
- no dedup, or dedup that loses first-seen order
- where dropped from the chunk filter, or same-field check off or stopping at the top level
- sums replaced with the first chunk or the max chunk; find_many_in keeping only the last page
- atomicity given a default
- line number added to the baseline key, occurrence index pinned to 0
- stale entries or new findings no longer failing the run
- --update-baseline dropping unscanned entries or keeping fixed ones
- helper exemption removed or matched by file name only
- raising the chunk size to 40,000 fails 3 of the 4 real-Postgres tests
- chunk_size bounds loosened or tightened by one on either side, the check removed, the max raised to 32,767, or chunk_size ignored in the range or the slice
- the chunked-field write guard removed, skipped for empty lists or for operator forms, or matching by substring
- the expression dropped from the baseline key, not whitespace-normalized, or a raw-SQL slot running to the end of the query
- the TypedDict skip removed, matching any x.TypedDict, taking the first argument, or missing fields= or typing_extensions
- _details_for_user_ids reverted to the raw filter, or chunks sent as tuples

Type

🆕 New Feature
🚄 Infrastructure

Caveats (if any)

Medium

update_many_in / delete_many_in over 5,000 values run as several statements
- Atomic only on a transaction handle; atomicity="caller_transaction" records that choice
- per_chunk_ok leaves earlier chunks applied if a later one fails
not_in cannot be chunked, so those sites need <> ALL($1::text[]) or a relation filter

Low

No group_by_in or take/skip/order/distinct: none of them survive a split
The 156 grandfathered findings still need migrating; the reset budget job is bounded by its 500-row batch
An identical expression re-added on the same field in the same function is the same finding, by design
Removing one of two same-field findings with the same expression in a function shifts the other's occurrence index
- The run then reports one new and one stale entry until --update-baseline
from x import NAME constants are not resolved across modules; no current finding needs it
- The check_batch_cost.py false positive was a spread of a local tuple constant, now counted as fixed
The 40,000-user integration test also passes on the pre-migration code, since the engine splits a plain find_many on its own; the migration is for consistency and for the cases it does not split
Every membership filter is reported, reads included, since the filter dict is usually built away from the call and the engine's find_many chunking is partial (Bug: findMany() with two large in filters with arrays of 32767 items each fails with Assertion violation, too many bind variables in prepared statement prisma/orm#21802)
misc / Run tests red is main's own: test_price_map_has_no_duplicate_keys failed on main until fix(cost-map): remove duplicate openrouter/perceptron/perceptron-mk1.5 entry #43273 (4d7aa89), which is newer than this branch's merge base
codecov/patch reads chunked_in.py at 61% because tests/unit runs only in the CircleCI unit job, started by the run-ci label
- Locally tests/unit/repositories/test_chunked_in.py covers it at 96% (52 passed); the two missed lines are the case _ arm of _logical_clauses
CodeQL py/mixed-returns notes (alerts 13011, 13012) on _as_clauses and _logical_clauses
- Every match arm returns, case _ included, so no path falls through to None; left as is since a push to appease a note resets the bot verdicts
ci/circleci: local_testing_part1 red is main's own: test_get_model_info_bedrock_cross_region_capability_parity fails at this tip and passes on the tip merged with main
- Fixed by fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows #42951 (4179860), which is newer than this branch's merge base
ci/circleci: local_testing_part2 red predates this PR: test_openai_stream_options_call_text_completion fails identically at the merge base 8694c3c, with 'MockValSer' object is not an instance of 'SchemaSerializer'
- Main's scheduled pipelines today pass it, and none of this PR's files are on that path
The Before proof leg ran at 7688f56, 305 commits behind the merge base 8694c3c
- Equivalent for both cases: no main commit carries the checker or the module
- The bind cap is the engine's, so a re-run at the merge base changes nothing

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Touches proxy spend metadata DB reads and adds broad CI rules for membership filters; chunked writes can be partially applied unless run in a transaction.

Overview
Adds CI enforcement and a repository helper so Prisma/SQL IN lists cannot silently exceed Postgres’s 32,767 bind-parameter limit.

A new litellm.repositories.chunked_in module exposes find_many_in, count_in, update_many_in, and delete_many_in, which dedupe values, run 5k-value chunks (with optional extra where), and merge results; writes require an explicit atomicity choice. Related Prisma table protocols were added for typing.

Code Quality now runs check_unbounded_in_lists.py, which AST-scans litellm/ and enterprise/ for unbounded Prisma "in"/"not_in" filters and runtime-spliced raw SQL IN ( patterns. ~156 existing sites are grandfathered in unbounded_in_baseline.txt; new or stale baseline entries fail the job.

One production migration: spend-log key metadata user lookup (_details_for_user_ids) uses find_many_in instead of a single find_many with user_id: {in: ...}. Unit, integration (40k rows), and checker tests cover the helper and CI behavior.

^{Reviewed by Cursor Bugbot for commit 9c7eae6. Bugbot is set up for automated code reviews on this repo. Configure here.}

9c7eae6 passes /live-pr-risk

greptile-apps · 2026-09-23T00:49:06Z

Confidence Score: 5/5

[High risk] Adds database query chunking and a CI check for unbounded SQL lists.

The PR appears safe to merge based on the current findings.

Summary

The PR adds chunked Prisma membership helpers, migrates user-detail recovery to use them, and makes the unbounded-list check enforce a baseline in CI.

Adds unit and database integration coverage for large lists and the migrated lookup.

_{Reviews (5) · Last reviewed commit: "ci: key an unbounded IN list finding by ..."}

ryan-crabbe-berri · 2026-09-23T00:54:44Z

@greptile re review

codecov · 2026-09-23T00:57:46Z

Codecov Report

❌ Patch coverage is 64.40678% with 21 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/repositories/chunked_in.py	61.11%	21 Missing ⚠️

📢 Thoughts on this report? Let us know!

ryan-crabbe-berri · 2026-09-23T01:04:11Z

@greptile re review

ryan-crabbe-berri · 2026-09-24T00:22:08Z

how do you deal with the fact that the direct sql function call might have an unbounded in but the place with the actual chunking logic living as an upstream caller ?

Postgres caps a prepared statement at 32,767 bind parameters and a membership filter binds one per value, so an IN list built from table data breaks once the table outgrows the cap. That is how the budget reset job froze every due budget (LIT-7535, #40564). check_unbounded_in_lists.py reports every Prisma "in" / "not_in" filter whose value has no fixed size and every raw SQL literal that splices a list in after "IN (", unless the line carries "# bounded-ok: <reason>". It only warns for now: the output is the inventory for RCA action item AI-1, and it exits 0.

An ALL_CAPS name imported or filled at runtime is as unbounded as any other, so a name now passes only when the module binds it once to a value of fixed size. Adds Final to the locals a loop does not forbid.

A module list bound once could still grow through append or extend, so a name now counts as fixed only when it is bound to a tuple, frozenset or constant. Trims the module docstring to what a reader needs.

…ded ones Add litellm.repositories.bounded_in: find_many_in, count_in, update_many_in and delete_many_in split a deduplicated value list into 5,000-value chunks, AND each chunk with the caller's where, run them in order (a transaction handle works) and combine the results. Writes take a required atomicity argument, and a where that already filters the chunked field is refused. check_unbounded_in_lists.py now fails CI on any finding missing from unbounded_in_baseline.txt and on any stale baseline entry, so the baseline only shrinks. Entries are keyed by path, enclosing scope, kind, field and occurrence, not line numbers. The helper module is exempt, a constant spread into a frozen tuple counts as fixed, and messages point at the helper for "in" and at an array parameter for "not_in" and raw SQL. A real-Postgres integration test shows a raw 40,000-value filter rejected for too many bind variables while the helpers handle it.

…k size The helper module is litellm.repositories.chunked_in, and its unit and integration tests, the checker's exemption path and its finding messages follow the new name. The `# bounded-ok` marker is unchanged. find_many_in, count_in, update_many_in and delete_many_in take a keyword-only chunk_size, defaulting to IN_LIST_CHUNK_SIZE (5,000). A value below 1 or above MAX_IN_LIST_CHUNK_SIZE (30,000) raises ValueError before any query, which leaves the rest of the filter headroom under Postgres's 32,767 bind-parameter cap.

…_iterable LIT014 (#42650) caps a comprehension at one for and one if clause. The four nested walks in the helper now chain their iterables instead, with the same order and results.

+    pass
+
+
+def _as_clauses(value: object) -> tuple[object, ...]:


+            return (value,)
+
+
+def _logical_clauses(clause: object) -> tuple[object, ...]:


codspeed · 2026-09-26T01:21:58Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_lint_unbounded_in_lists (9c7eae6) with main (4179860)¹}

No successful run was found on main (2ef3250) during the generation of this report, so 4179860 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

ryan-crabbe-berri · 2026-09-26T01:33:35Z

bugobt run

ryan-crabbe-berri · 2026-09-26T01:33:42Z

@greptileai re review

…ists _details_for_user_ids reads users through find_many_in instead of a raw "in" filter, so its lookup stays under the bind-parameter cap for any number of recovered keys. Up to 5,000 ids it still sends one find_many with the same where dict, and a PrismaError from any chunk is still logged and treated as no details. The helper now sends each chunk as a list, so a chunked filter equals the dict a hand-written call would send and a migrated call site's existing assertions keep passing. The site's baseline entry is gone.

The dict passed as the field map of TypedDict("Name", {...}), or as its fields= keyword, names fields: an "in" or "notIn" key there is a type, not a filter. Only that dict is skipped, for TypedDict, typing.TypedDict and typing_extensions.TypedDict; a filter nested in a field value or passed to any other call is still reported. The two types/proxy/management_endpoints/team_endpoints.py entries leave the baseline, which is now 156.

Chunks run one after another, so an update that sets the chunked field can move a row into a later chunk, which updates it again and counts it twice: values ["old", "new"] with chunk_size=1 and data={"id": "new"} does exactly that. update_many_in now raises ChunkedFieldWriteError before any query when data has the chunked field as a top-level key, in any form, including Prisma operators such as {"set": ...}.

…and how to clear it It now says what is reported, the three ways to clear a finding, and how the baseline and --update-baseline work, in 11 lines. The per-shape detail lives in the tests.

A baseline key of path, scope, kind, field and occurrence let a PR delete a baselined filter and add a different unbounded one on the same field in the same function, and the new one took over the old key. The key now also carries the filtered expression's source, whitespace-normalized (the Prisma value, or a raw-SQL `IN (...)` slot), so that swap reads as one new and one stale entry and fails the run. The same expression re-added in the same function is still the same finding. Every baseline entry is rewritten in the new form; the 156 findings are unchanged, and only occurrence indexes renumber where one field had several different expressions.

ryan-crabbe-berri · 2026-09-26T02:20:45Z

@greptileai

ryan-crabbe-berri · 2026-09-26T02:58:18Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9c7eae6. Configure here.}

mateo-berri

LGTM. Thanks!

ryan-crabbe-berri requested a review from a team September 23, 2026 00:46

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread tests/code_coverage_tests/check_unbounded_in_lists.py Outdated

Comment thread tests/code_coverage_tests/check_unbounded_in_lists.py

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread tests/code_coverage_tests/check_unbounded_in_lists.py Outdated

Comment thread tests/code_coverage_tests/check_unbounded_in_lists.py Outdated

ryan-crabbe-berri changed the title ~~ci: warn on SQL IN lists with no written bound~~ ci: fail on new unbounded SQL IN lists and add a Prisma chunking helper Sep 25, 2026

ryan-crabbe-berri added 6 commits September 25, 2026 17:47

ci: decide a constant IN list by its module binding, not its casing

e6ce81f

An ALL_CAPS name imported or filled at runtime is as unbounded as any other, so a name now passes only when the module binds it once to a value of fixed size. Adds Final to the locals a loop does not forbid.

ci: only a frozen module value makes an IN list constant

2429930

A module list bound once could still grow through append or extend, so a name now counts as fixed only when it is bound to a tuple, frozenset or constant. Trims the module docstring to what a reader needs.

refactor: flatten chunked_in's stacked comprehensions with chain.from…

0b2e80a

…_iterable LIT014 (#42650) caps a comprehension at one for and one if clause. The four nested walks in the helper now chain their iterables instead, with the same order and results.

ryan-crabbe-berri force-pushed the litellm_lint_unbounded_in_lists branch from f9a4a26 to 0b2e80a Compare September 26, 2026 01:08

github-advanced-security Bot found potential problems Sep 26, 2026

View reviewed changes

Comment thread litellm/repositories/chunked_in.py

pass

def _as_clauses(value: object) -> tuple[object, ...]:

Comment thread litellm/repositories/chunked_in.py

return (value,)

def _logical_clauses(clause: object) -> tuple[object, ...]:

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread tests/code_coverage_tests/check_unbounded_in_lists.py

Comment thread litellm/repositories/chunked_in.py

ryan-crabbe-berri added 5 commits September 25, 2026 19:11

docs: cut the unbounded IN list checker's docstring to what it flags …

8b58092

…and how to clear it It now says what is reported, the three ways to clear a finding, and how the baseline and --update-baseline work, in 11 lines. The per-shape detail lives in the tests.

cursor Bot reviewed Sep 26, 2026

View reviewed changes

mateo-berri added the run-ci label Sep 26, 2026

mateo-berri approved these changes Sep 26, 2026

View reviewed changes

mateo-berri merged commit 96c008f into main Sep 26, 2026
143 of 164 checks passed

mateo-berri deleted the litellm_lint_unbounded_in_lists branch September 26, 2026 20:40

mateo-berri mentioned this pull request Sep 30, 2026

fix(proxy): backport #40541, #43642, and #43656 to stable/1.103.x for v1.103.2 #43897

Merged

13 tasks done

		return (value,)


		def _logical_clauses(clause: object) -> tuple[object, ...]:

Conversation

ryan-crabbe-berri commented Sep 23, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (7688f56)

CI on a new unbounded list

40,000 ids through Prisma

After (9c7eae6)

CI on a new unbounded list

40,000 ids through Prisma

Tests and mutation check

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

greptile-apps Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

ryan-crabbe-berri commented Sep 23, 2026

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ryan-crabbe-berri commented Sep 23, 2026

Uh oh!

ryan-crabbe-berri commented Sep 24, 2026

Uh oh!

codspeed Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

ryan-crabbe-berri commented Sep 26, 2026

Uh oh!

ryan-crabbe-berri commented Sep 26, 2026

Uh oh!

Uh oh!

Uh oh!

ryan-crabbe-berri commented Sep 26, 2026

Uh oh!

ryan-crabbe-berri commented Sep 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryan-crabbe-berri commented Sep 23, 2026 •

edited by mateo-berri

Loading

Before (`7688f56`)

After (`9c7eae6`)

greptile-apps Bot commented Sep 23, 2026 •

edited

Loading

codecov Bot commented Sep 23, 2026 •

edited

Loading

codspeed Bot commented Sep 26, 2026 •

edited

Loading