fix: tiktoken cache nonroot offline

[milan-berri]

Summary

This PR fixes a regression in tiktoken cache handling for non-root + offline environments and aligns both eager and lazy loading with the same offline-safe behavior.

Today, non-root Docker images with --network=none can fail on startup because tiktoken attempts to download the cl100k_base vocab into an empty cache directory, instead of using the pre-bundled vocab that LiteLLM ships.

---

Background

LiteLLM bundles the cl100k_base vocab at:

litellm/litellm_core_utils/tokenizers/

Historically, default_encoding.py set:

TIKTOKEN_CACHE_DIR = <bundled tokenizers dir>
encoding = tiktoken.get_encoding("cl100k_base")

This guaranteed:

• Offline-safe: tiktoken reads the bundled vocab, no network needed.
• Non-root-safe: as long as the directory is readable, no writes are required.

PR #19449 changed this for non-root images by introducing:

is_non_root = os.getenv("LITELLM_NON_ROOT", "").lower() == "true"
if not os.access(filename, os.W_OK) and is_non_root:
    filename = "/tmp/tiktoken_cache"
    os.makedirs(filename, exist_ok=True)
TIKTOKEN_CACHE_DIR = CUSTOM_TIKTOKEN_CACHE_DIR or filename

This solved “can’t write into read-only site-packages” by redirecting non-root to /tmp/tiktoken_cache, but it also introduced a regression:

• /tmp/tiktoken_cache starts empty.
• In offline or --network=none environments, tiktoken cannot download the vocab into that empty dir → startup fails.
• This affects all non-root images since v1.81.9 unless CUSTOM_TIKTOKEN_CACHE_DIR is manually pointed back at the bundled dir.

Separately, PR #19774 fixed lazy loading by making __getattr__ use _get_default_encoding(), which assumes that default_encoding.py has correctly set TIKTOKEN_CACHE_DIR to a local cache. That PR does not address the /tmp behavior itself; it just ensures lazy loading respects whatever default_encoding.py configured.

---

Fix

This PR simplifies and hardens the cache selection logic in default_encoding.py:

1. Always resolve the bundled tokenizer directory:

filename = "<...>/litellm_core_utils/tokenizers"

2. Use it as the default cache dir, unless explicitly overridden:

custom_cache_dir = os.getenv("CUSTOM_TIKTOKEN_CACHE_DIR")
if custom_cache_dir:
    os.makedirs(custom_cache_dir, exist_ok=True)
    cache_dir = custom_cache_dir
else:
    cache_dir = filename

os.environ["TIKTOKEN_CACHE_DIR"] = cache_dir

Key points:

• Default (no env):

  • TIKTOKEN_CACHE_DIR → bundled tokenizers dir.
  • tiktoken reads the pre-bundled vocab (offline-safe, works for root and non-root).

• When CUSTOM_TIKTOKEN_CACHE_DIR is set:

  • We os.makedirs(custom_cache_dir, exist_ok=True) so the directory always exists.
  • TIKTOKEN_CACHE_DIR → that custom path.
  • This gives users full control if they want a writable cache (e.g. custom images), without changing the default behavior.

• Removed logic:

  • We no longer special-case LITELLM_NON_ROOT or redirect to /tmp/tiktoken_cache by default.
  • Non-root images now behave the same as root images by default, using the shipped vocab, which is sufficient for read-only filesystems.

---

Why this approach

• Restores offline behavior:
  • Non-root + --network=none uses the bundled vocab again; no network fetch is attempted.
• Keeps non-root safe:
  • Reading from a pre-bundled file does not require write access, even under read-only site-packages.
• Keeps lazy loading correct:
  • _get_default_encoding() (and PR #19774) already rely on default_encoding.py to set TIKTOKEN_CACHE_DIR to a local cache.
  • With this change, both eager and lazy paths now share the same offline-safe default and the same override semantics.
• Minimal surface area:
  • All tiktoken cache decisions remain centralized in one place (default_encoding.py).
  • No extra copying, no per-env heuristics beyond a single optional override env var.

---

Tests

Updated tests in tests/test_default_encoding_non_root.py:

1. test_default_encoding_uses_bundled_tokenizers_by_default

   • Sets LITELLM_NON_ROOT="true" with no CUSTOM_TIKTOKEN_CACHE_DIR.
   • Reloads default_encoding.
   • Asserts:
     • TIKTOKEN_CACHE_DIR is set.
     • Its path contains "tokenizers" (i.e., points at the bundled directory).

2. test_custom_tiktoken_cache_dir_override

   • Creates a temporary path tmp_path / "tiktoken_cache".
   • Sets CUSTOM_TIKTOKEN_CACHE_DIR to that path.
   • Reloads default_encoding.
   • Asserts:
     • TIKTOKEN_CACHE_DIR == custom_dir.
     • os.path.isdir(TIKTOKEN_CACHE_DIR) → the directory was created.

These tests verify both:

• The default offline/non-root behavior (no more /tmp redirect).
• The override and mkdir semantics for CUSTOM_TIKTOKEN_CACHE_DIR.

---

Impact

• Fixes the regression where non-root images with no network fail due to tiktoken trying to download into an empty /tmp cache.
• Keeps lazy loading behavior from #19774 intact, since TIKTOKEN_CACHE_DIR is now consistently initialized to a local cache before any get_encoding call.
• Maintains an explicit escape hatch (CUSTOM_TIKTOKEN_CACHE_DIR) for advanced setups, while keeping the default path simple and offline-safe.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🐛 Bug Fix

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Error		Mar 13, 2026 0:57am

[greptile-apps]

Greptile Summary

This PR fixes a regression introduced in #19449 where non-root Docker images with --network=none could fail on startup because TIKTOKEN_CACHE_DIR was redirected to an empty /tmp/tiktoken_cache instead of the pre-bundled vocab directory. The fix simplifies default_encoding.py to always point TIKTOKEN_CACHE_DIR at the bundled tokenizers/ directory by default, with CUSTOM_TIKTOKEN_CACHE_DIR as the sole explicit override.

Key changes:

• Removes the LITELLM_NON_ROOT + os.access write-check heuristic and the /tmp/tiktoken_cache fallback entirely — non-root images now use the same offline-safe bundled path as root images
• CUSTOM_TIKTOKEN_CACHE_DIR still works as an escape hatch; os.makedirs is called to ensure the custom path exists before use
• Tests are rewritten to use importlib.reload for true behavior verification; test_custom_tiktoken_cache_dir_override patches tiktoken.get_encoding to avoid implicit network calls when pointed at an empty directory
• _reload_default_encoding helper now clears both TIKTOKEN_CACHE_DIR and CUSTOM_TIKTOKEN_CACHE_DIR before each reload, addressing the test isolation issue flagged in the previous review
• Cleanup reload at the end of test_custom_tiktoken_cache_dir_override ensures default_encoding.encoding is a real tiktoken Encoding for subsequent tests

Confidence Score: 5/5

• Safe to merge — minimal, well-scoped change that restores intended offline behavior without breaking any existing interfaces.
• The production code change is a clean simplification: it removes a special-case heuristic and restores the pre-regression default. No public API surface is changed. The CUSTOM_TIKTOKEN_CACHE_DIR escape hatch is preserved. The previously-flagged test isolation and mock issues have all been addressed. The one remaining minor style point (missing cleanup reload in the first test) does not affect correctness.
• No files require special attention.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/default_encoding.py	Removes the LITELLM_NON_ROOT /tmp fallback; always defaults TIKTOKEN_CACHE_DIR to the bundled tokenizers directory unless CUSTOM_TIKTOKEN_CACHE_DIR overrides it. Logic is clean and offline-safe.
tests/test_default_encoding_non_root.py	Tests rewritten to reload the module for real behavior verification. The custom-cache test correctly patches tiktoken.get_encoding to stay offline-safe. The first test calls real tiktoken.get_encoding without a mock — intentional but environment-sensitive.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[default_encoding.py module loads] --> B[Resolve bundled tokenizers dir via importlib.resources]
    B --> C{CUSTOM_TIKTOKEN_CACHE_DIR set?}
    C -- Yes --> D[os.makedirs custom_cache_dir, exist_ok=True]
    D --> E[cache_dir = custom_cache_dir]
    C -- No --> F[cache_dir = bundled tokenizers dir]
    E --> G[os.environ TIKTOKEN_CACHE_DIR = cache_dir]
    F --> G
    G --> H[import tiktoken]
    H --> I[tiktoken.get_encoding cl100k_base with retry logic]
    I --> J[encoding = tiktoken Encoding object]

    style F fill:#d4edda,color:#000
    style D fill:#fff3cd,color:#000

Loading

_{Last reviewed commit: a6f8389}

[greptile-apps]

Module state not restored after mocked reload

After test_custom_tiktoken_cache_dir_override completes and the with patch(...) block exits, default_encoding.encoding is left as a MagicMock() in sys.modules. The monkeypatch fixture restores the environment variables, but it does not reload the module back to a clean state.

litellm/_lazy_imports.py calls from litellm.litellm_core_utils.default_encoding import encoding lazily inside _get_default_encoding(), which executes the import at call time. If any test that runs after this one (in the same process) invokes _get_default_encoding() before _reload_default_encoding is called again, it will receive a MagicMock instead of a real tiktoken Encoding.

To be safe, consider adding a module teardown reload at the end of the test (after the assertions) so the module is restored to a clean state regardless of test execution order:

    # After assertions — restore module to a clean, real-encoding state
    monkeypatch.delenv("TIKTOKEN_CACHE_DIR", raising=False)
    importlib.reload(default_encoding)

[greptile-apps]

No module-state cleanup after first test

test_default_encoding_uses_bundled_tokenizers_by_default calls _reload_default_encoding, which reloads the module and sets default_encoding.encoding to a freshly-loaded tiktoken Encoding. There is no cleanup reload at the end of this test function to restore the module to its original state (unlike test_custom_tiktoken_cache_dir_override, which explicitly reloads at lines 55–57).

In practice this is low-risk because the encoding is still a real tiktoken Encoding object loaded from the bundled tokenizers, so downstream consumers of default_encoding.encoding won't receive a broken value. However, if a subsequent test calls _get_default_encoding() in _lazy_imports.py before any other reload, it will use the already-cached _default_encoding (set when litellm was first imported) and won't be affected. The concern is if default_encoding.encoding is ever accessed directly (not via the lazy cache), it may differ from the original import-time state.

For symmetry and defensive consistency, consider adding a cleanup reload at the end of this test, as done in test_custom_tiktoken_cache_dir_override:

    assert "tokenizers" in cache_dir

    # Restore module to original state
    monkeypatch.delenv("TIKTOKEN_CACHE_DIR", raising=False)
    monkeypatch.delenv("CUSTOM_TIKTOKEN_CACHE_DIR", raising=False)
    importlib.reload(default_encoding)

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}