Repository navigation
fix(token_counter): release the GIL for HuggingFace counts and cap exact counting per string - #40186
Conversation
…act counting per string Both proxy token counting endpoints already count in a worker thread, but the HuggingFace tokenizer's encode holds the GIL for the whole call, so a 600k-token count on a Claude model still froze the event loop for up to 0.8 s and every other request with it. Count through encode_batch_fast, which releases the GIL, and tokenize at most TOKEN_COUNTER_MAX_EXACT_CHARS characters of any one string (default 4,000,000), scaling the exact count of that prefix by the string's length above it so the largest payloads stay bounded.
…itellm_lit_5546_count_tokens_offload
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Greptile SummaryThis PR prevents large token counts from blocking proxy event loops. It moves relevant proxy counting paths onto a dedicated, bounded worker-thread limiter, uses the HuggingFace batch encoder, caps exact counting through evenly distributed sampling, and fixes custom tokenizer loading. The latest change scopes the limiter to each event loop so processes using several loops do not share incompatible AnyIO synchronization state Confidence Score: 5/5The PR appears safe to merge, with the previous blocking findings fixed or withdrawn and no new actionable issue found The latest limiter change gives every event loop a compatible dedicated limiter while retaining bounded concurrency among proxy requests on that loop. Tests cover shared-pool isolation, concurrency bounds, multi-loop completion, sampling, and event-loop responsiveness. The prefix-sampling, worker-pool starvation, parameter typing, and coarse test type findings are fixed; the synchronous SDK finding was withdrawn after the proxy boundary was clarified
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/token_counter.py | Adds capped token-count extrapolation, GIL-friendly HuggingFace encoding, and a dedicated per-event-loop offload limiter |
| litellm/proxy/proxy_server.py | Offloads custom tokenizer loading and endpoint token counting from the event loop |
| litellm/proxy/utils.py | Moves failure-path usage counting onto the bounded token-count worker pool |
| litellm/router.py | Offloads router pre-call token estimation through the dedicated limiter |
| litellm/router_utils/pre_call_checks/io_token_rate_limit_check.py | Moves deployment ITPM input estimation off the event loop |
| litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py | Offloads prompt-caching eligibility counts while preserving routing behavior |
| tests/test_litellm/litellm_core_utils/test_token_counter.py | Covers event-loop responsiveness, sampling bounds, limiter isolation, configuration, and multiple event loops |
Reviews (10): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Devin e2e QATested at: 486af36 Shared setup. One proxy per leg, started from a checkout of the commit named in the heading with model_list:
- model_name: anthropic-badkey
litellm_params:
model: anthropic/claude-fable-5
api_key: sk-ant-bogus-key-for-fallback-leg
general_settings:
master_key: sk-1234The bogus key makes the Anthropic CountTokens call fail, so Payloads were built from repository source files: Each case runs leg() {
NAME=$1; URL=$2; PAYLOAD=$3; OUT=runs/$NAME; rm -rf $OUT; mkdir -p $OUT
( sleep 0.5; curl -s -o $OUT/count_response.json -w '%{http_code} %{time_total}\n' -X POST http://127.0.0.1:28283$URL \
-H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' --data-binary @$PAYLOAD > $OUT/count_status.txt ) &
CPID=$!; i=0
while kill -0 $CPID 2>/dev/null; do
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' http://127.0.0.1:28283/health/liveliness > $OUT/$i &
i=$((i+1)); sleep 0.02
done
wait
echo "count_tokens: http_code time_total = $(cat $OUT/count_status.txt)"
echo "response: $(head -c 160 $OUT/count_response.json)"
echo -n "liveliness during $NAME: "; python3 stats.py $OUT
}The last case restarts the proxy with Before (7da6fe5)Captured at the merge base with baseline
/utils/token_counter, Claude tokenizer, messages
/utils/token_counter, Claude tokenizer, one 2.4M-character prompt string, default cap
/v1/messages/count_tokens, provider count fails, local Claude tokenizer fallback
/utils/token_counter, same prompt string, TOKEN_COUNTER_MAX_EXACT_CHARS=100000
After (486af36)baseline
/utils/token_counter, Claude tokenizer, messages
/utils/token_counter, Claude tokenizer, one 2.4M-character prompt string, default cap
/v1/messages/count_tokens, provider count fails, local Claude tokenizer fallback
/utils/token_counter, same prompt string, TOKEN_COUNTER_MAX_EXACT_CHARS=100000
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…les instead of the prefix A string above TOKEN_COUNTER_MAX_EXACT_CHARS was counted from its first cap characters and scaled, so a string whose start tokenizes unlike its end got a skewed count, and that count reaches fallback billing when the provider sends no usage. The estimate now tokenizes 16 evenly spaced samples that together total the cap and scales their sum by the string's length, keeping the same bound on work while tracking the whole string
The failure hook's usage estimate and the project ITPM reservation both called litellm.token_counter inline on the event loop, so a large request that failed or hit the limiter stalled the gateway the same way the count_tokens endpoints did. Both now run through asyncify. The loop-lag probe the existing tests used moves into a shared helper that warms the tokenizer first, and two new tests fail when either count runs inline
|
bugbot run |
…f large counts cannot exhaust memory
|
bugbot run |
…acity limiter Replace the thread semaphore around HuggingFace encodes with an anyio CapacityLimiter applied at every offloaded count site through offload_token_count, so waiting counts no longer hold slots in the shared 40-thread pool and inline counts on the event loop never block on the bound. Rename TOKEN_COUNTER_MAX_CONCURRENT_HF_ENCODES to TOKEN_COUNTER_MAX_CONCURRENT_COUNTS
|
bugbot run |
| def count_tokens(text: str) -> int: | ||
| enc: Final = tokenizer_json["tokenizer"].encode(text) | ||
| return len(enc.ids) | ||
| return len(tokenizer.encode_batch_fast([text])[0]) |
There was a problem hiding this comment.
Synchronous HuggingFace token-counter callers now bypass concurrency control, so concurrent SDK counts can again multiply tokenizer memory usage
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
tiktoken, the default path, already releases the GIL and runs uncapped from SDK threads; proxy counts are bounded by the limiter and per-string sampling
There was a problem hiding this comment.
You’re right—I conflated the synchronous SDK API with the proxy paths this PR is changing. The proxy call sites use offload_token_count, so HuggingFace counts are bounded by the per-worker limiter; the default synchronous tiktoken path is already GIL-free and intentionally remains uncapped for SDK callers. I’m withdrawing this as a blocking finding.
A single process-wide anyio.CapacityLimiter wakes waiters with the asyncio.Event of whichever loop created it, so a fifth loop in another thread (asyncio.run per request under Celery, gunicorn sync workers, or run_async_function from function_with_fallbacks) waited forever once four counts were in flight, and building it at import time raised AsyncLibraryNotFoundError on anyio below 4.2. The limiter now lives in a RunVar and is created on first use inside the running loop. TOKEN_COUNTER_MAX_CONCURRENT_COUNTS reads from the environment like TOKEN_COUNTER_MAX_EXACT_CHARS, and the proxy's tokenizer lookup runs on the shared thread pool so a Hub download no longer holds a count slot.
…itellm_lit_5546_count_tokens_offload
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit c1ca963. Configure here.
261aa2f
into
litellm_internal_staging
1 participant
TLDR
Problem this solves:
itpmreservation all ran inline on the event loop, on any tokenizer/utils/token_counterloaded a deployment's custom HuggingFace tokenizer on the loop with a keyword tokenizers rejects, so every custom count logged an error and loaded the tokenizer twiceHow it solves it:
TOKEN_COUNTER_MAX_CONCURRENT_COUNTSslots per worker process (default 4, settable from 1 to 256 in the proxy's environment) instead of the 40-thread pool every other threaded call shares, so a burst of large counts is bounded per worker and never starves the shared poolcompletion()with fallbacks, Celery, gunicorn) finishes instead of waiting forever on a wake-up sent to another loop, and the import works on anyio releases before 4.2TOKEN_COUNTER_MAX_EXACT_CHARScharacters per string, sampled evenly, scaled by lengthUser Flow
Before: one whole-conversation count freezes the gateway for every other developer while it runs
anthropic/claude-*models, which a local count tokenizes with the Claude tokenizer); the team also has Bedrock deployments withitpmlimits andoptional_pre_call_checks: [enforce_model_rate_limits]POST https://litellm-domain/v1/messages/count_tokens(the Anthropic SDK'smessages.count_tokens), sending the system prompt, tools, and messages; the provider's count fails, the gateway counts locally, and the app gets200 {"input_tokens": 618344}back after about 0.7 sGET https://litellm-domain/health/livelinessprobe, which normally answers in about 1 ms, takes up to 80 ms, and every Claude Code developer'sPOST https://litellm-domain/v1/messageswaits just as long before the gateway even starts on itPOST https://litellm-domain/utils/token_countergets200 {"total_tokens": 614612}after about 0.7 s and holds the probe for up to 660 msPOST https://litellm-domain/v1/chat/completionswith the 600k-token prompt through a deployment whose provider key is bad) comes back401from Anthropic or403from Bedrock after about 1 s, and the probe stalls for up to 690 ms (Anthropic) or 260 ms (Bedrock) while the gateway counts that failed request's tokens for its spend recordPOST https://litellm-domain/v1/messagesseven times in 45 s, and the probe stalls for up to 690 ms on every attempt while that failed request's tokens are countedPOST https://litellm-domain/v1/chat/completionsthrough theitpm-limited Bedrock deployment holds the probe for up to 220 ms while the gateway counts the prompt to reserve it against the limit, before the provider is even calledoptional_pre_call_checks: [prompt_caching]on, everyPOST https://litellm-domain/v1/chat/completionsthrough a Claude deployment holds the probe for two whole-prompt counts, one while picking the deployment and one after the responsePOST https://litellm-domain/utils/token_counterfor a deployment withmodel_info.custom_tokenizerloads the tokenizer from the Hub cache on the loop on every call, about 0.1 s each, and logsError creating pretrained tokenizerbefore loading it a second timePOST https://litellm-domain/utils/token_counterat the same moment are tokenized one after another under the GIL, and the probe stalls behind each of themAfter: the same counts run alongside the other traffic, which keeps answering in about 1 ms
POST https://litellm-domain/v1/messages/count_tokensand gets the same200 {"input_tokens": 618344}back after about 0.7 sGET https://litellm-domain/health/livelinessprobe keeps answering in about 1 ms, and every Claude Code developer'sPOST https://litellm-domain/v1/messagesstarts right awayPOST https://litellm-domain/utils/token_countergets the same200 {"total_tokens": 614612}with the probe flat at about 1 msPOST https://litellm-domain/v1/chat/completionsstill comes back401or403after about 1 s, and the probe stays at about 1 ms while the failed request's tokens are counted401on every attempt, with the probe at about 1 ms while each attempt's tokens are countedPOST https://litellm-domain/v1/chat/completionsthrough theitpm-limited deployment is still reserved and still rejected with429 ITPM limit=10, current usage=541553when the limit is too small, with the probe at about 1 ms while the prompt is countedTOKEN_COUNTER_MAX_EXACT_CHARSin the proxy's environment200 {"total_tokens": 501831}within 1.0 to 1.6 s, each worker counting at most four at a time (TOKEN_COUNTER_MAX_CONCURRENT_COUNTSin the proxy's environment moves that number); with two workers eight still run at once and the proxy grows by about the same 1.4 GB as before, while sixteen at once peak at 1.6 GB instead of twice that because the second eight wait for a slot, and the probe stays answered (worst 124 ms) instead of stallingRelevant issues
No GitHub issue. Reported by a customer through support (Pylon #7623 and #7411). Pylon #7314 was the tiktoken side of the same stall and was fixed in #37697; tiktoken releases the GIL, so a Bedrock model id (which selects tiktoken) never stalled the count_tokens routes, which is why the Bedrock cases below are controls. Docs: BerriAI/litellm-docs#1259. Remaining inline counts outside these paths (the context-management editors, the streaming chunk builder's no-usage fallback, the compression interception hook, the Anthropic passthrough's interrupted-stream re-tokenize, the A2A usage counter, and the SDK-only
acount_tokensandencode) are tracked in LIT-7190; keeping a custom tokenizer loaded across/utils/token_countercalls is LIT-7192Linear ticket
Resolves LIT-5546
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Every leg below ran twice: Before at 21ae29b, the PR's merge base with
origin/litellm_internal_staging, and After at c1ca963, the tip of this branch. The proxy for each side is started from a checkout of that commit withPYTHONPATHpointed at it, and the script asserts the importedlitellm.__file__sits inside that tree before booting (theloaded:line at the top of each side). Same config, same payloads, same order on both sides. The box carried a load average of 16 to 30 from other jobs throughout, which thebaselinelegs (the probe loop with nothing in flight) put on record for each sideThe rig runs twice per side: once with one uvicorn worker, where a blocked event loop is visible on every leg because the only worker is the one counting, and once with two workers (
--num_workers 2), the multi-pod shape customers run, where a stalled worker is hidden from single-request probes by the free one and shows up when both workers are busy (the burst leg and the over-cap strings)Config:
bedrock-claudeis a tiktoken control: this AWS account gets 403 from Bedrock CountTokens, so/utils/token_counterfalls back to local counting, and a Bedrock model id selects tiktoken.bedrock-badcredandbedrock-itpmcarry a bogus Bedrock API key (a per-deploymentapi_keyis sent as the bearer token) so the same fallback runs and a chat completion fails upstream with a real403;bedrock-itpmadds anitpmlimit far above the prompt so the deployment reservation counts the prompt before the call, andbedrock-itpm-tiny(itpm: 10) shows that check is live by rejecting the prompt.anthropic-badkeymakes the Anthropic count fail so/v1/messages/count_tokenstakes the local fallback on the Claude tokenizer, and makes a chat completion fail upstream with a real401so the gateway counts the failed request's tokens itself.custom-tokis a deployment with acustom_tokenizerpulled from the HuggingFace hub withauth_token: null(thecreate_pretrained_tokenizer(token=)path),custom-tok-badnames a repository that does not exist. The twoanthropic-pcdeployments withprompt_cachinginoptional_pre_call_checksexercise the check whose two whole-prompt counts this PR moves to a worker thread.disable_cooldowns: truekeeps the bad-credential deployments in rotation after their first rejectionPayloads are 2.5 MB JSON transcripts built from repository source files: 323 alternating user and assistant messages, 2,403,245 characters, about 615k tokens on the Claude tokenizer.
anthropic_utils.jsonandbedrock_utils.jsoncarry them as{"model": ..., "messages": [...]}for/utils/token_counter;badkey_anthropic.jsonis the Anthropic Messages shape{"model": "anthropic-badkey", "system": <6 KB>, "tools": [<6 tools>], "messages": [...]}for/v1/messages/count_tokens;badkey_chat.jsonis{"model": "anthropic-badkey", "messages": [...], "max_tokens": 16}for/v1/chat/completions;prompt_anthropic_utils.jsonis the same text as onepromptstring;bedrock_badcred_utils.json,bedrock_badcred_chat.json,bedrock_itpm_chat.json, andbedrock_itpm_tiny_chat.jsonare the same payloads with the model swapped.small_hf_utils.jsonis a 2,000,000-character prompt for the burst,custom_utils.jsona 300,000-character prompt,custom_bad_utils.jsonan 11-character one,pc_chat.jsona chat completion with a 3,600-token system prompt markedcache_control: {"type": "ephemeral"}andReply with just the word OKas the user turnBefore every side the tokenizers were loaded once and the failure path warmed once so no leg times a first load:
for m in anthropic-claude bedrock-claude bedrock-badcred custom-tok anthropic-claude anthropic-claude anthropic-badkey; do curl -s -o /dev/null -X POST http://127.0.0.1:41681/utils/token_counter -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d "{\"model\": \"$m\", \"prompt\": \"warm up the tokenizer\"}"; doneandfor m in anthropic-badkey bedrock-badcred bedrock-itpm; do curl -s -o /dev/null -X POST http://127.0.0.1:41681/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d "{\"model\": \"$m\", \"messages\": [{\"role\": \"user\", \"content\": \"warm up the failure path\"}], \"max_tokens\": 5}"; doneEach API leg runs
leg, which posts the payload once and hits/health/livelinessevery 20 ms for as long as the request is in flight, then prints the request's status and time (the label readscount_tokens:for every timed request, the chat completion legs included) and the p50/p90/p99/max of the probes taken during it.burstfires n copies of a request at once, samples the proxy process tree's RSS every 100 ms while they run, and probes the same way.scenariois a single timed curltree_rss_mbsumsps -o rss=over the proxy pid and its children,scenarioiscurl -s -o body -w 'status=%{http_code} total=%{time_total}s'on the route plus the first 400 bytes of the body, andstats.pyreads the probe files and printsprobes= ok= p50 p90 p99 max mean. The prompt caching leg postspc_chat.jsontwice and prints the content plus theusagecache fields of each answerThe Claude Code legs drive the real client interactively. Claude Code 2.1.267 runs in tmux behind a small recorder (
python3 cc/recorder.py 44741 cc/recorder.log 41681, which forwards to the proxy and logsMETHOD path body_bytes=Nper request) asenv -u ANTHROPIC_API_KEY ANTHROPIC_BASE_URL=http://127.0.0.1:44741 ANTHROPIC_AUTH_TOKEN=sk-1234 ANTHROPIC_MODEL='bedrock-claude[1m]' ANTHROPIC_SMALL_FAST_MODEL='bedrock-claude[1m]' CLAUDE_CONFIG_DIR=cc/config claude. The first leg typesReply with just the word OK, waits for the answer, then types/context, which makes Claude Code count the pieces of its context with 32 smallcount_tokensrequests (the largest 74 KB) while the script probes/health/livelinessevery 20 ms for 25 s, then prints the/contextpane and takes a screenshot. The second leg starts a fresh session onanthropic-badkey(a revoked key), pastescc/paste_full.txt(2,405,645 bytes, the same text as the 2.4 MB curl legs) as one message and presses Enter: Claude Code sendsPOST /v1/messageswith a 2,515,689-byte body, Anthropic answers 401, Claude Code retries with backoff, and the gateway counts each failed attempt's prompt on the Claude tokenizer for its spend record, so the probe loop (45 s) sees whether those counts stall the loop. Claude Code never sends a whole-transcriptcount_tokenson this build, so this is the route by which a Claude Code user's oversized conversation reaches local countingOne worker, Before (21ae29b) and After (c1ca963)
baseline, no request in flight (4 s of probes)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Claude tokenizer, 323 messages (2.5 MB)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Claude tokenizer, one 2,403,245-character prompt string
Before (21ae29b):
After (c1ca963):
/v1/messages/count_tokens, provider count fails, local Claude tokenizer fallback
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Bedrock model id, tiktoken fallback (control)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Bedrock model id with a bad credential, tiktoken fallback (control)
Before (21ae29b):
After (c1ca963):
/v1/chat/completions rejected by Anthropic, failed request's tokens counted for its spend record
Before (21ae29b):
After (c1ca963):
/v1/chat/completions rejected by Bedrock, failed request's tokens counted for its spend record
Before (21ae29b):
After (c1ca963):
/v1/chat/completions through a Bedrock deployment with an itpm limit, prompt counted for the reservation
Before (21ae29b):
After (c1ca963):
the itpm check is live on this rig (a 10-token limit rejects the same prompt)
Before (21ae29b):
After (c1ca963):
eight 2 MB /utils/token_counter requests at once, Claude tokenizer, memory sampled every 100 ms
Before (21ae29b):
After (c1ca963):
/utils/token_counter, a deployment's custom HuggingFace tokenizer (300 KB prompt)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, a custom tokenizer identifier that does not exist
Before (21ae29b):
After (c1ca963):
prompt caching pre-call check, the same cached prompt twice
Before (21ae29b):
After (c1ca963):
baseline again at the end of the side
Before (21ae29b):
After (c1ca963):
Claude Code, /context in an interactive session
Before (21ae29b):
After (c1ca963):
Claude Code, a 2.4 MB conversation Anthropic rejects, retried
Before (21ae29b):
After (c1ca963):
Two workers, Before (21ae29b) and After (c1ca963)
Same legs with
--num_workers 2. Two workers hide a single request's stall from the probes (the free worker answers them), so the API legs read close on both sides and the difference shows where both workers are counting: the burst leg, where the Before side's probes reach 6.6 s (p90 1.8 s) while the After side's stay under 70 ms, and the over-cap strings belowbaseline, no request in flight (4 s of probes)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Claude tokenizer, 323 messages (2.5 MB)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Claude tokenizer, one 2,403,245-character prompt string
Before (21ae29b):
After (c1ca963):
/v1/messages/count_tokens, provider count fails, local Claude tokenizer fallback
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Bedrock model id, tiktoken fallback (control)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, Bedrock model id with a bad credential, tiktoken fallback (control)
Before (21ae29b):
After (c1ca963):
/v1/chat/completions rejected by Anthropic, failed request's tokens counted for its spend record
Before (21ae29b):
After (c1ca963):
/v1/chat/completions rejected by Bedrock, failed request's tokens counted for its spend record
Before (21ae29b):
After (c1ca963):
/v1/chat/completions through a Bedrock deployment with an itpm limit, prompt counted for the reservation
Before (21ae29b):
After (c1ca963):
the itpm check is live on this rig (a 10-token limit rejects the same prompt)
Before (21ae29b):
After (c1ca963):
eight 2 MB /utils/token_counter requests at once, Claude tokenizer, memory sampled every 100 ms
Before (21ae29b):
After (c1ca963):
/utils/token_counter, a deployment's custom HuggingFace tokenizer (300 KB prompt)
Before (21ae29b):
After (c1ca963):
/utils/token_counter, a custom tokenizer identifier that does not exist
Before (21ae29b):
After (c1ca963):
prompt caching pre-call check, the same cached prompt twice
Before (21ae29b):
After (c1ca963):
baseline again at the end of the side
Before (21ae29b):
After (c1ca963):
Claude Code, /context in an interactive session
Before (21ae29b):
After (c1ca963):
Claude Code, a 2.4 MB conversation Anthropic rejects, retried
Before (21ae29b):
After (c1ca963):
Over-cap strings, two workers
The payload is a 4,600,000-character prompt built from repository source files, above
TOKEN_COUNTER_MAX_EXACT_CHARS(4,000,000), asbig_hf_utils.json({"model": "anthropic-claude", "prompt": ...}) andbig_hf_messages.json(the same text as one user message for/v1/messages/count_tokensonanthropic-badkey, so the provider count fails and the local Claude tokenizer counts it). Before counts the whole string exactly and stalls both workers for 2.7 s and 3.0 s; After tokenizes 16 evenly spaced samples totalling 4,000,000 characters and scales, landing 0.2% under the exact count, with no probe above 6 msBefore (21ae29b):
After (c1ca963):
Burst, two workers, eight then sixteen then eight at once
Once HuggingFace encodes release the GIL they run in parallel, so a burst of large counts multiplies the tokenizer's working memory by however many run at once. c1ca963 runs every offloaded count under a per-event-loop
anyio.CapacityLimiter(TOKEN_COUNTER_MAX_CONCURRENT_COUNTS)(4 slots per worker process by default), separate from the 40-thread limiter every other threaded call shares. The eight-at-once burst on the merge base is in the two-worker legs above (probes p50 in the seconds while the loop counts them one after another, requests answered in 1.5 s to 17 s); this is the same burst on the tip, plus sixteen at once and eight again, on the same two-worker proxyFor comparison, the first offloaded build of this branch (d202885, counts on the shared 40-thread pool with nothing else bounding them) took the eight-at-once burst to
proxy tree rss peak during burst = 1753 MB (+1584 MB), after = 1591 MBwith the requests all answered inside 1.3 s; the limiter trades some of that parallelism (eight requests now finish in 1.2 s to 2.1 s, sixteen in 3.0 s to 7.8 s) for a bounded peakReading the two sides
Type
🐛 Bug Fix
Caveats (if any)
Medium
itpmcounts. The Claude tokenizer fix applies to deployments whose model is a bare Claude id after the provider prefix (Anthropic direct, Vertex, custom tokenizers)TOKEN_COUNTER_MAX_EXACT_CHARStrades the estimate back for worker timeTOKEN_COUNTER_MAX_CONCURRENT_COUNTS=1brings back the serial memory profile while keeping the loop freelitellm.token_counter,litellm.completion's own pre-call and failure counts) get the GIL-free encode with no limiter: eight threads counting 2.4 MB each now run on all cores (9.6 s of CPU in 2.4 s of wall time against 5.4 s in 5.8 s at the merge base) and peaked at +1200 MB against +145 MB. Only the proxy's offloaded paths are boundedLow
TOKEN_COUNTER_MAX_CONCURRENT_COUNTSper worker queuesencode_batch_fastfails the count, andcreate_pretrained_tokenizer's identifier fallback for such objects is gone with itmax(lags) < took / 4, a timing bound a very slow CI runner could trip; it passed on three consecutive runs here with the merge base failing 8 of 9 unambiguouslylitellm.acount_tokensandlitellm.utils.encodestill count inline, andencodeis uncapped (LIT-7190)offload_token_count, so at the merge base they fail by ImportError rather than by the assertion; the lag assertions themselves fail there when the import is stubbed (8 of 9 legs)test_completion_novita_ai,test_azure_gpt_4o_with_tool_call_and_response_format,test_openai_max_retries_0,test_openai_o_series_max_retries_0, each a provider answering with nochoices), and the same tests are red onlitellm_internal_staging's own pipeline; none of them is a required checkosv-scanis red onsmol-toml1.6.1 inui/litellm-dashboard/package-lock.json(GHSA-7w5x-hrqm-74c2, a dev dependency), which this PR does not touch; it is red on staging's own runs for the same reason and is not a required checkFinal Attestation
Note
Medium Risk
Changes rate-limit reservations, spend on failures, and token estimates for oversized prompts; very long inputs become sampled approximations without signaling in responses.
Overview
Stops large HuggingFace and proxy-side token counts from blocking the asyncio event loop, and bounds how much text is fully tokenized per string.
Offloading: Adds
offload_token_count, which runs sync counters on worker threads behind a per-event-loopanyio.CapacityLimiter(TOKEN_COUNTER_MAX_CONCURRENT_COUNTS, default 4). Proxy/router paths that used to count inline or on the shared thread pool now use this helper—including/utils/token_counter, project/deployment ITPM reservations, failure-hook usage lifting, prompt-caching checks, and router pre-call token checks. HuggingFace counting switches toencode_batch_fastso encoding can release the GIL; custom Hub tokenizers load withtoken=instead of the rejectedauth_tokenkeyword.Caps: Strings longer than
TOKEN_COUNTER_MAX_EXACT_CHARS(default 4M chars) are estimated from 16 evenly spaced samples and scaled by length. New tests assert the loop stays responsive, concurrency/env limits are honored, and extrapolation stays within ~1% of exact counts.Reviewed by Cursor Bugbot for commit c1ca963. Bugbot is set up for automated code reviews on this repo. Configure here.