initialize tiktoken environment at import time to support offline usage

[milan-berri]

Relevant issues

Fixes #19768, #19852

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix

Changes

• initialize tiktoken environment at import time to support offline usage

Testing

To reproduce the issue and verify the fix:

Click to expand reproduction script

#!/bin/bash

set -e

echo "Step 1: Clean environment setup"
source venv_dev/bin/activate

echo "Step 2: Remove tiktoken and cache completely"
pip uninstall tiktoken -y 2>/dev/null || echo "tiktoken not installed"

TEMP_DIR=$(python -c "import tempfile; print(tempfile.gettempdir())")
echo "Removing tiktoken cache from: $TEMP_DIR"
rm -rf "$TEMP_DIR"/data-gym-cache* 2>/dev/null || true
rm -rf "$TEMP_DIR"/*tiktoken* 2>/dev/null || true

echo "Step 3: Install fresh tiktoken"
pip install tiktoken

echo "Step 4: Verify clean state - no fix applied"
grep -n "default_encoding" venv_dev/lib/python3.13/site-packages/litellm/__init__.py || echo "No fix found - clean state confirmed"

echo "Step 5: Block tiktoken URL"
echo "127.0.0.1 openaipublic.blob.core.windows.net" | sudo tee -a /etc/hosts > /dev/null

echo "Step 6: Test original issue"
echo "Testing lazy loading disabled (should fail):"
python -c "import os; os.environ['LITELLM_DISABLE_LAZY_LOADING'] = '1'; import litellm; print('UNEXPECTED SUCCESS')" 2>&1 || echo "EXPECTED FAILURE - reproduced the issue"

echo "Testing lazy loading enabled (should fail):"
python -c "import litellm; print('UNEXPECTED SUCCESS')" 2>&1 || echo "EXPECTED FAILURE - reproduced the issue"

echo "Step 7: Apply fix"
sed -i '' '/dotenv.load_dotenv()/a\
from litellm.litellm_core_utils import default_encoding  # noqa: F401' venv_dev/lib/python3.13/site-packages/litellm/__init__.py

echo "Step 8: Verify fix applied"
grep -n "default_encoding" venv_dev/lib/python3.13/site-packages/litellm/__init__.py

echo "Step 9: Test fix works"
echo "Testing lazy loading disabled (should work):"
python -c "import os; os.environ['LITELLM_DISABLE_LAZY_LOADING'] = '1'; import litellm; print('SUCCESS - fix works')" 2>&1

echo "Testing lazy loading enabled (should work):"
python -c "import litellm; print('SUCCESS - fix works')" 2>&1

echo "Step 10: Cleanup"
sudo sed -i '' '/openaipublic.blob.core.windows.net/d' /etc/hosts

echo "Reproduction complete!"

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
litellm	Error		Jan 27, 2026 8:09pm

[CLAassistant]

All committers have signed the CLA.

[milan-berri]

Attaching a screenshot with the outputs of the above testing script.

[Sameerlite]

Works

[TylerBrabham]

Doesn't this PR break lazy loading since it's loading tiktoken on initial import?

I don't think this PR is necessary when combined with #19774

[Sameerlite]

@TylerBrabham You might be right, @AlexsanderHamir Will take a look at this once