test: finish the non-proxy half of tests/test_litellm

[devin-ai-integration]

TLDR

Problem this solves:

• Two unit tests still reach the network (huggingface.co, a fake hostname)
• Key-gated real-provider tests and empty folders still sit in tests/test_litellm outside proxy/
• Docs, Makefile and GHA shards still point at legacy tests/test_litellm paths that no longer exist

How it solves it:

• test_encoding_and_decoding serves the llama2 tokenizer from the in-memory hub already used by HUB_TOKENIZER_SCRIPT, and asserts the hub was actually hit
• test_ahealth_check_failure_masks_raw_request_headers targets a reserved loopback port and asserts Authorization is present and masked unconditionally
• R100 moves of interactions/, the llms/ provider files and test_compression.py into tests/llm_translation/; deletes log.txt, ocr/, passthrough/
• Empties the two legacy llms shards and the misc shard's dead paths in test-unit.yml, drops the ocr/** filter in test-rust.yml
• make test-unit now runs tests/unit tests/test_litellm; docs point at tests/unit; stale path sentences in test docstrings are deleted

Out of scope, untouched: everything under tests/test_litellm/proxy/ and its references, synthetic path strings used as test inputs, litellm/ code, and the cost map JSON files (sibling PR #43273). A test_price_map_has_no_duplicate_keys red is inherited from main.

Commits: 1) rename-only (git diff --stat -M shows 100% renames), 2) offline tests, 3) CI, Makefile, docs, 4) stale docstring references and the two moved files' "Run with" lines, 5) Greptile follow-ups (test_databricks_e2e.py sys.path depth after the move, CONTRIBUTING example pointing at the proxy test that still lives in tests/test_litellm/proxy/).

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Network audit (sys.addaudithook on socket.getaddrinfo and socket.connect, loopback ignored), same two tests before and after:

# before (main)
RESULT: 2 passed
RECORDED HOSTS:
huggingface.co
invalid-endpoint-that-will-fail.com

# after (this branch), plus test_token_counter_uses_the_tokenizer_of_each_model_family_and_of_a_custom_tokenizer
RESULT: 3 passed
NET AUDIT HOSTS: <none>

Caveat: the hub subprocess runs python -I, which ignores PYTHONPATH and sitecustomize, so the audit hook does not instrument that child. Its isolation is proven by the test itself: huggingface_hub.set_client_factory is pinned to an httpx.MockTransport and the test asserts result["requested"] == ["hf-internal-testing/llama-tokenizer"].

Mutations (each applied, run, then reverted):

# 1. tokenizer factory not set (llama2 falls back to tiktoken)
FAILED tests/unit/litellm_core_utils/test_token_counter.py::test_encoding_and_decoding
  assert len(result["tokens"]) != len(encode(model="gpt-3.5-turbo", text=sample_text))

# 2. health check records raw (unmasked) headers
FAILED tests/unit/litellm_core_utils/test_health_check_helpers.py::test_ahealth_check_failure_masks_raw_request_headers
  assert 'Bearer dapi-test-key-1234567890abcdef' != 'Bearer dapi-test-key-1234567890abcdef'

# 3. Authorization dropped from recorded headers (used to pass silently behind the old `if`)
FAILED tests/unit/litellm_core_utils/test_health_check_helpers.py::test_ahealth_check_failure_masks_raw_request_headers
  assert "Authorization" in headers

Moves and node ids: 58 old node ids map one to one onto 58 new node ids (pytest --collect-only -q on each side, diffed after path substitution). pytest --collect-only -q tests/llm_translation collects 2853 tests with no errors.

After cleanup, tests/test_litellm outside proxy/ holds only __init__.py, conftest.py, readme.md, test_conftest.py.

Coverage and layout checks:

python .github/scripts/assert_ci_coverage.py            OK
python .github/scripts/assert_ci_coverage.py --shards   OK
python .github/scripts/assert_ci_coverage.py --slices   OK
pytest tests/unit/test_package_layout.py tests/unit/test_assert_ci_coverage.py tests/unit/test_circleci_path_filter.py
  102 passed
make -n test-unit  ->  uv run pytest tests/unit tests/test_litellm -x -vv -n 4

Stale path grep (rg 'tests/test_litellm' -g '!tests/test_litellm/**'): every remaining hit is either under tests/test_litellm/proxy/..., a reference to the still-existing tests/test_litellm/conftest.py, or a synthetic path string inside test_assert_ci_coverage.py, test_circleci_path_filter.py, test_cost_map_guard.py, all explicitly out of scope.

Link to Devin session: https://app.devin.ai/sessions/9d2761dd5003414faa8919818f5b3e78
Open in Devin Desktop: https://app.devin.ai/desktop/session/9d2761dd5003414faa8919818f5b3e78?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ yuneng-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Reorganizes test directory structure and updates documentation.

The PR appears safe to merge; no outstanding findings remain.

Summary

The PR relocates non-proxy tests, makes two tests run without external network access, and updates CI and contributor paths. Since the previous review, it corrected the documented proxy-test example and the Databricks direct-run import path.

_{Reviews (2) · Last reviewed commit: "fix: correct databricks e2e sys.path dep..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_test_litellm_nonproxy_cleanup (5b9c5a4) with main (d08746f)¹}

Footnotes

1. No successful run was found on main (4eb340a) during the generation of this report, so d08746f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai