docs(count_tokens): document TOKEN_COUNTER_MAX_EXACT_CHARS and off-loop local counting

[devin-ai-integration]

What changed

docs/proxy/config_settings.md gains the TOKEN_COUNTER_MAX_EXACT_CHARS row in the environment variable table, and docs/count_tokens.md explains that local token counting on the proxy runs off the event loop and that strings longer than that many characters (default 4,000,000, roughly a million tokens) are counted from 16 evenly spaced samples adding up to that many characters and scaled to the full length

Why

Pairs with the LiteLLM change for LIT-5546, where one very large count_tokens payload could stall every other request on the gateway. The setting is new in that PR and needs to be discoverable from the settings reference and the token counting page

Linear ticket

Resolves LIT-5546

Link to Devin session: https://app.devin.ai/sessions/ea3d050c61f64334a5474e411af182a9
Open in Devin Desktop: https://app.devin.ai/desktop/session/ea3d050c61f64334a5474e411af182a9?variant=devin
Requested by: @mateo-berri

---

Note

Low Risk
Documentation-only changes with no runtime or configuration behavior modified in this PR.

Overview
Documents proxy-side local token counting safeguards that pair with the runtime fix for large count_tokens payloads (LIT-5546).

docs/count_tokens.md now states that local counting on the proxy runs off the event loop (worker thread), caps concurrent counts per worker via TOKEN_COUNTER_MAX_CONCURRENT_COUNTS (default 4, with queuing beyond that), and for strings longer than TOKEN_COUNTER_MAX_EXACT_CHARS (default 4,000,000) uses 16 evenly spaced samples scaled to full length instead of full exact tokenization.

docs/proxy/config_settings.md adds reference rows for both environment variables so operators can discover and tune them.

^{Reviewed by Cursor Bugbot for commit 6e8e368. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
litellm	Ready	Preview	Sep 10, 2026 1:26am UTC

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6e8e368. Configure here.}

[mateo-berri]

LGTM