fix(deps): upgrade tiktoken to 0.14.0

[silencedoctor]

TLDR

Problem this solves:

• LiteLLM currently locks tiktoken 0.12.0 and permits installs back to 0.8.0
• Newer GPT-5 family names such as gpt-5.6 are not mapped by 0.12.0, so LiteLLM falls back to cl100k_base
• The 0.13.0 and 0.14.0 releases include BPE performance work, broader GPT-5 mappings, and current Python wheels

How it solves it:

• Raises the base dependency floor to tiktoken>=0.14.0,<1.0
• Refreshes uv.lock to resolve tiktoken 0.14.0
• Adds a regression test through LiteLLM's token counter using a GPT-5.6 model and a text sample where o200k_base and cl100k_base differ

Relevant issues

Fixes #39716

Related: #26193, #37697

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The Python dependency checks and focused unit tests pass locally
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review and received a confidence score of at least 4/5

Screenshots / Proof of Fix

The proof is command-line based because this is a dependency and token-counting change

With the frozen lockfile:

$ uv lock --check
Resolved 455 packages in 25ms

$ uv run --frozen --no-sync python -c 'import importlib.metadata, tiktoken; from litellm.litellm_core_utils.token_counter import token_counter; text="你好，世界。"; print("tiktoken=" + importlib.metadata.version("tiktoken")); print("encoding=" + tiktoken.encoding_for_model("gpt-5.6").name); print("token_counter=" + str(token_counter(model="gpt-5.6", text=text)))'
tiktoken=0.14.0
encoding=o200k_base
token_counter=4

The focused regression suite passes:

$ uv run --frozen --no-sync pytest tests/test_litellm/litellm_core_utils/test_token_counter.py -q
115 passed, 1 skipped in 6.05s

The same wheel and mapping were verified with uv on Python 3.10, 3.11, 3.12, 3.13, and 3.14

Type

Bug Fix

Changes

pyproject.toml raises the runtime floor, uv.lock records the 0.14.0 sdist and wheels, and test_token_counter.py protects the GPT-5 mapping behavior

Caveats

make check reached Prisma generation but stopped at the dashboard bootstrap because this machine has npm 11.6.2 while the dashboard requires npm >=11.10.0. Python-side lock, import, token-counting, and Ruff checks pass

The upstream fork-dependency guard rejects uv.lock changes from cross-repository PRs by design. This PR includes the lock refresh because the requested upgrade is incomplete without it; if the guard blocks the workflow, a maintainer will need to apply the same lockfile change from a branch in the canonical repository

Final Attestation

• The tests check the new GPT-5 mapping and the dependency resolves to the intended release

[codspeed]

Merging this PR will improve performance by 64.46%

⚡ 4 improved benchmarks
✅ 27 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	test_token_counter_long_content	1,729 µs	601.3 µs	×2.9
⚡	test_token_counter_multi_turn	589.3 µs	387.4 µs	+52.12%
⚡	test_token_counter_with_tools	421.2 µs	295.7 µs	+42.44%
⚡	test_token_counter_raw_text	189.8 µs	161.7 µs	+17.4%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing silencedoctor:litellm_upgrade_tiktoken_014 (7bc6275) with litellm_internal_staging (c8635ec)}

[silencedoctor]

@greptileai Please review this dependency upgrade

[greptile-apps]

Greptile Summary

This PR upgrades the minimum tiktoken version to 0.14.0 and refreshes the lockfile so newer GPT-5 models use their upstream encoding mappings

• Adds a focused GPT-5.6 token-counting regression test that distinguishes o200k_base from the prior fallback
• Keeps the dependency capped below 1.0
• Includes wheel artifacts for all supported Python versions and major platforms

Confidence Score: 5/5

The dependency upgrade appears safe to merge, with consistent lock metadata and focused regression coverage

No actionable failures remain. The supported Python range has matching wheels, production usage relies on compatible public APIs, and the new test verifies the intended model mapping behavior

Important Files Changed

Filename	Overview
pyproject.toml	Raises the tiktoken runtime dependency floor to 0.14.0 without conflicting with supported Python versions
tests/test_litellm/litellm_core_utils/test_token_counter.py	Adds a meaningful regression test through LiteLLM's production token-counting path for GPT-5.6
uv.lock	Consistently resolves tiktoken 0.14.0 and records artifacts covering the repository's supported Python range

_{Reviews (1): Last reviewed commit: "fix(deps): upgrade tiktoken to 0.14.0" | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!