fix(token_counter): stop large token counts from blocking the proxy event loop

[devin-ai-integration]

TLDR

Problem this solves:

• Low-entropy text stalls token counting for tens of seconds
• The count runs on the proxy event loop, blocking everything
• Liveness and readiness probes time out, replicas drop

How it solves it:

• Encode long text in bounded chunks, not one call
• Caps tiktoken's quadratic merge cost, keeps counting linear
• Async paths now count in a worker thread, off the loop
• Chunk-size setting is bounded, so misconfiguration cannot break counting
• Solves two of the ticket's three call sites, mitigates the third

User Flow

Before: a developer sending a large multimodal agent request gets it served, but the gateway stops answering anything else for the better part of a minute, and the replica gets pulled out of rotation

1. They send POST https://litellm-domain/v1/messages with a Claude model and a large body whose text contains long runs of a single repeated character (dot leaders and padding from extracted form text)
2. The request sits there with no response for tens of seconds
3. Every other request against the same replica, including a plain GET https://litellm-domain/health/liveliness, hangs for that whole window instead of returning in milliseconds, and some are refused outright once the connection backlog fills
4. The platform team sees the pod fail its liveness and readiness probes with context deadline exceeded, CPU pinned at exactly 1.0 vCPU, no OOM and no restart, and unrelated in-flight calls surface bogus litellm.Timeout: Connection timed out. Timeout passed=30.0, time taken=0.001 seconds errors
5. POST https://litellm-domain/utils/token_counter on a body of the same shape stalls the replica the same way
6. Their only workaround is litellm_settings: disable_token_counter: true, which gives up token-based context-window pre-filtering and makes POST https://litellm-domain/v1/messages/count_tokens return 503 whenever the provider's own count API fails

After: the same request is served promptly and the replica keeps answering everything else while it is in flight

1. They send the same POST https://litellm-domain/v1/messages with the same body
2. The request comes back in seconds, with the same token totals as before
3. GET https://litellm-domain/health/liveliness answers in about a millisecond throughout, so probes pass and the replica stays in rotation
4. POST https://litellm-domain/utils/token_counter on the same body returns in under a tenth of a second and never stalls concurrent traffic
5. No spurious 30s timeout errors appear on concurrent requests, and no disable_token_counter workaround is needed

Relevant issues

Linear ticket

Resolves LIT-5426

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: a live proxy on port 4010 with router_settings: {enable_pre_call_checks: true} and one direct Anthropic deployment, so the router's context-window pre-call check runs on the request path:

model_list:
  - model_name: anthropic-sonnet-4-6
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  enable_pre_call_checks: true

general_settings:
  master_key: sk-1234

PYTHONPATH=. python litellm/proxy/proxy_cli.py --config lit5426_config.yaml --port 4010
python -c "import json; print(json.dumps({'model':'anthropic-sonnet-4-6','max_tokens':4,'messages':[{'role':'user','content':'A'*400000}]}))" > payload2.json
python -c "import json; print(json.dumps({'model':'gpt-3.5-turbo','prompt':'A'*400000}))" > payload3.json

Cases 2 and 3 each send one request in the background and fire 150 liveness probes at 0.2s intervals while it is in flight:

curl -s -o resp.json -w 'request: %{http_code} in %{time_total}s\n' --max-time 180 -X POST "$URL" -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' --data @"$PAYLOAD" & req=$!
for i in $(seq 1 150); do (curl -s -o /dev/null -w "liveliness[$i]: %{http_code} in %{time_total}s\n" --max-time 45 http://localhost:4010/health/liveliness > "probes/$i" 2>&1) & sleep 0.2; done
wait $req

Before (cde1344)

Case 1: SDK token counting (the ticket's own repro)

1. Ran:

LITELLM_LOCAL_MODEL_COST_MAP=True python -c "
import time, litellm
t0 = time.perf_counter()
n = litellm.token_counter(model='us.anthropic.claude-sonnet-4-6', messages=[{'role': 'user', 'content': 'A' * 262144}])
print(f'{n} tokens in {time.perf_counter() - t0:.1f}s')"

2. Output:

32775 tokens in 18.8s

Case 2: liveness probes while one large POST /v1/messages is in flight

1. Ran the shared case commands against http://localhost:4010/v1/messages with payload2.json
2. Output:

case2: 200 in 46.104470s
liveliness[1]: 200 in 41.384411s
liveliness[2]: 200 in 41.184932s
liveliness[3]: 200 in 40.963598s
...
liveliness[150]: 000 in 7.787908s

3. 129 of 150 probes eventually answered 200 with a median of 25.27s and a max of 41.38s; the other 21 were refused outright once the accept backlog filled behind the blocked loop
4. The request itself hit real Anthropic and was billed:

{"model":"anthropic-sonnet-4-6","id":"msg_011CeEndVd8gwwGSFEHK7Had","type":"message","role":"assistant","content":[{"type":"text","text":"A."}],"stop_reason":"max_tokens","usage":{"input_tokens":133343,"output_tokens":4}}

Case 3: liveness probes while one large POST /utils/token_counter is in flight

1. Ran the shared case commands against http://localhost:4010/utils/token_counter with payload3.json
2. Output:

case3: 200 in 38.921233s
liveliness[1]: 200 in 38.915943s
liveliness[2]: 200 in 38.703705s
liveliness[3]: 200 in 38.486415s
...
liveliness[150]: 000 in 7.782811s

3. 128 of 150 probes answered 200 with a median of 22.93s and a max of 38.92s, and 22 were refused
4. Response:

{"total_tokens":50000,"request_model":"gpt-3.5-turbo","model_used":"gpt-3.5-turbo","tokenizer_type":"openai_tokenizer","error":false}

After (db68877)

Case 1: SDK token counting (the ticket's own repro)

1. Ran the same command as above
2. Output:

32775 tokens in 0.1s

Case 2: liveness probes while one large POST /v1/messages is in flight

1. Ran the shared case commands against http://localhost:4010/v1/messages with payload2.json
2. Output:

case2: 200 in 4.823487s
liveliness[1]: 200 in 0.102209s
liveliness[2]: 200 in 0.001002s
liveliness[3]: 200 in 0.001014s
...
liveliness[150]: 200 in 0.001108s

3. All 150 probes answered 200, median 0.0010s, max 0.1022s
4. The request hit real Anthropic again and reported the same input token total as Before, so nothing about the count changed:

{"model":"anthropic-sonnet-4-6","id":"msg_011CeEpLgWjZiiYFmwsU3V7M","type":"message","role":"assistant","content":[{"type":"text","text":"A."}],"stop_reason":"max_tokens","usage":{"input_tokens":133343,"output_tokens":4}}

Case 3: liveness probes while one large POST /utils/token_counter is in flight

1. Ran the shared case commands against http://localhost:4010/utils/token_counter with payload3.json
2. Output:

case3: 200 in 0.098131s
liveliness[1]: 200 in 0.002550s
liveliness[2]: 200 in 0.000841s
liveliness[3]: 200 in 0.000957s
...
liveliness[150]: 200 in 0.000993s

3. All 150 probes answered 200, median 0.0010s, max 0.0042s
4. Response reports the same 50000 tokens as Before:

{"total_tokens":50000,"request_model":"gpt-3.5-turbo","model_used":"gpt-3.5-turbo","tokenizer_type":"openai_tokenizer","error":false}

Type

🐛 Bug Fix

Review notes

What this covers, and what it does not

LIT-5426 names three call sites. Two are now genuinely off the event loop and this PR solves them: the router context-window pre-call check, and the local fallback behind /utils/token_counter and /v1/messages/count_tokens. The proof above measures exactly those: 150 of 150 liveness probes answering at a 1.0ms median where the merge base managed 129 of 150 at a 25.27s median with 21 refused outright.

The third, the cost-calculator fallback that runs when a response carries no usage, is mitigated rather than solved. _async_success_handler_body calls _success_handler_helper_fn directly, with no executor and no thread hop, so that count still runs on the loop. Chunking takes it from over five minutes (killed) to 0.49s on a 2.4MB single-character payload, which is bounded and linear but not zero. That 0.49s is a bench number for the encode itself, not a proxy measurement: unlike the two cases above, I did not instrument that path against a live proxy.

I did not take the third because moving it off-loop means making the async logging path async through completion_cost, which reaches well beyond token counting and belongs in its own change

Why the encode moved off the loop

Chunking on its own only shortens the stall. The whole chunk loop still ran on the event loop thread, so a big enough payload kept monopolizing it, which is the ticket's actual premise. Measured on this machine, a single encode of "A" * 200000 blocks the loop for 8.26s: a competing coroutine's largest scheduling gap is 8263ms. Running the same encode through a worker thread leaves the max gap at 2.66ms and lets the competitor run 587k times, which confirms tiktoken releases the GIL for its Rust encode and that a thread offload genuinely helps here

So the async paths now count off-loop, through the repo's existing asyncify helper. /utils/token_counter awaits asyncify(litellm.token_counter), resolved at call time so the public seam callers and tests already patch keeps working, and the router's async_get_healthy_deployments counts off-loop and passes the result into _pre_call_checks rather than letting it count inline. _pre_call_checks keeps its own lazy inline count for the sync entry points, but the async caller sets skip_inline_token_count, so a count that fails off-loop is never retried on the loop; it just leaves the deployments unfiltered, which is what a failed count already did. For the same reason the pre-count resolves each deployment's context window exactly the way _pre_call_checks does, skipping ones it cannot map rather than letting the first unmappable deployment hide a limit declared by a later one. Sync callers such as completion_cost are unchanged: there is no loop to protect there, and chunking already bounds their cost

Why the chunk size is bounded on both ends

The lower bound closes more than a wrong number. A configured -1 produced no chunks at all and silently reported zero tokens, and a zero count is not cosmetic: every downstream consumer of it, the context-window pre-call filter and the budget and rate-limit accounting, then sees a free request, so a single mistyped environment variable quietly stopped the proxy enforcing its limits. The clamp closes that as well as the performance hole

The chunk size still needs a ceiling even with the offload, because the cost is linear in the chunk size: on a 2.4MB single-character payload it is 0.28s at 512, 0.49s at 1024, 1.8s at 4096 and 13.3s at 16384. TIKTOKEN_ENCODE_CHUNK_SIZE_CHARS is therefore clamped to [1, 4096]. Zero used to raise ValueError from range() and a negative value produced no chunks at all, silently reporting zero tokens and quietly disabling every context-window, trimming and accounting decision downstream. Out-of-range and unparseable values now warn and fall back to 1024

Caveats (if any)

• Chunking drifts ~0.06% on natural text over 1KB
• Cost-calculator fallback still counts on the loop, bounded to ~0.5s
• Sync callers still count on the calling thread
• Chunk sizes above 4096 are clamped, with a warning
• Anthropic-native image blocks still break counting, ticketed separately

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/e1ccb43093544b83b0bebcf711acd91b
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR bounds tiktoken encoding work and moves token counting off asynchronous proxy and router event-loop paths.

• Splits tiktoken input into bounded chunks and validates the configurable chunk size.
• Offloads /utils/token_counter fallback counting to a worker thread.
• Precomputes router context-window token counts off-loop, preserves limits from later resolvable deployments, and prevents failed counts from being retried inline.
• Adds regression coverage for chunk-size validation, asynchronous counting, heterogeneous deployments, and failed off-loop counts.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported event-loop counting paths, invalid chunk-size handling, and heterogeneous-deployment fallback have been addressed.

Important Files Changed

Filename	Overview
litellm/constants.py	Defines a bounded token-encoding chunk size, resolving the previously reported invalid-value behavior.
litellm/litellm_core_utils/env_utils.py	Adds integer environment parsing with inclusive range validation and safe fallback.
litellm/litellm_core_utils/token_counter.py	Encodes tiktoken input in bounded chunks to avoid pathological single-call merge costs.
litellm/proxy/proxy_server.py	Runs fallback token counting in a worker thread rather than on the proxy event loop.
litellm/router.py	Precomputes context-window counts off-loop, scans deployments independently for declared limits, and prevents inline recounting after failure.

_{Reviews (3): Last reviewed commit: "fix(token_counter): stop large token cou..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 97.82609% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/router.py	96.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing devin/lit-5426-fix (db68877) with litellm_internal_staging (e07a712)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (1d7f675) during the generation of this report, so e07a712 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

@greptileai re-review at 8c007b7: counts now run off the event loop in a worker thread, and the chunk-size setting is bounded and validated

[yassin-berriai]

@greptileai re-review at db68877: the off-loop count no longer falls back to an inline on-loop recount, and unmappable deployments no longer hide a declared limit

[tin-berri]

Foundational and billing-adjacent (token counting feeds context-window pre-filtering and cost estimation) so dispatched a deep review that empirically re-tested the claims against real tiktoken rather than trusting the PR's own numbers. Findings: the chunk-size clamp [1, 4096] with fallback-to-1024 genuinely fixes two real pre-existing bugs — chunk_size=0 previously raised ValueError from range(), and a negative value silently produced zero tokens via an empty range, which would have quietly disabled every downstream context-window/trimming/accounting decision — both edge cases are now clamped correctly and tested. The off-loop async offload (via the existing asyncify helper) and the new skip_inline_token_count flag are sound: no double-counting (input_tokens is seeded once, the inline branch only fires when the precomputed value is None), and a failed/None off-loop count falls through to the exact same 'return the full unfiltered deployment list' path the pre-existing exception handler already used — not a new degradation mode. One nuance worth a note: the disclosed ~0.06% chunking drift isn't strictly guaranteed to over-count — empirical testing found ~2% of cases under-count by 1-2 tokens due to tiktoken's regex pre-tokenizer splitting differently at a chunk boundary. The magnitude is immaterial for budget/context-window enforcement, but the PR's caveat could be a touch more precise about direction not being guaranteed. Two minor non-blocking nits: get_router_model_info is now called twice per deployment in the pre-call-check path, and the PR's own body honestly discloses a third _pre_call_checks-adjacent call site (the cost-calculator success handler) that remains on-loop, bounded only by chunking (~0.5s) — worth a tracked follow-up if one doesn't exist. Proof-of-fix is dramatic and real (live proxy, 46s->4.8s with actual Anthropic billing, all 150 liveness probes passing vs most timing out/refused before). CI green aside from the long-running benchmarks job.

[yassin-berriai]

Heads-up: this moves test_token_counter_multi_turn about -10% on CodSpeed instruction count. Expected, chunking costs instructions. Full detail on LIT-5426