Skip to content

litellm_fix_mapped_tests_core: fix test isolation and mock injection issues#20208

Closed
shin-bot-litellm wants to merge 1 commit intomainfrom
fix/mapped-tests-core-scientific-notation-and-mocks
Closed

litellm_fix_mapped_tests_core: fix test isolation and mock injection issues#20208
shin-bot-litellm wants to merge 1 commit intomainfrom
fix/mapped-tests-core-scientific-notation-and-mocks

Conversation

@shin-bot-litellm
Copy link
Contributor

Problem

Four tests in litellm_mapped_tests_core were failing:

  1. test_register_model_with_scientific_notation - KeyError due to test isolation issues
  2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path
  3. test_send_email_missing_api_key - Real API calls despite mocking
  4. test_stream_transformation_error_sync - Mock not effective, real API called

Solution

test_register_model_with_scientific_notation

  • Use unique model name to avoid conflicts with other tests
  • Clear LRU caches before test to prevent stale data
  • Clean up model_cost entry after test

test_search_uses_registry_credentials

  • Use patch.object() on the actual base_llm_http_handler instance
  • String-based patching for instance methods can fail; direct object patching is more reliable

test_send_email_missing_api_key

  • Directly inject mock HTTP client into logger instance
  • This bypasses any caching issues that could cause the fixture mock to be ineffective

test_stream_transformation_error_sync

  • Patch litellm.completion directly instead of the handler module litellm reference
  • This ensures the mock is effective regardless of import order

Regression

These tests were affected by LRU caching added in #19606 and HTTP client caching.

Failing Tests Link

https://app.circleci.com/pipelines/github/BerriAI/litellm/56737/workflows/9aa8a62f-0d73-4f49-a084-aa5408142a69/jobs/1140414/tests

…issues

## Problem
Four tests in litellm_mapped_tests_core were failing:
1. test_register_model_with_scientific_notation - KeyError due to test isolation issues
2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path
3. test_send_email_missing_api_key - Real API calls despite mocking
4. test_stream_transformation_error_sync - Mock not effective, real API called

## Solution

### test_register_model_with_scientific_notation
- Use unique model name to avoid conflicts with other tests
- Clear LRU caches before test to prevent stale data
- Clean up model_cost entry after test

### test_search_uses_registry_credentials
- Use patch.object() on the actual base_llm_http_handler instance
- String-based patching for instance methods can fail; direct object patching is more reliable

### test_send_email_missing_api_key
- Directly inject mock HTTP client into logger instance
- This bypasses any caching issues that could cause the fixture mock to be ineffective

### test_stream_transformation_error_sync
- Patch litellm.completion directly instead of the handler module's litellm reference
- This ensures the mock is effective regardless of import order

## Regression
These tests were affected by LRU caching added in #19606 and HTTP client caching.
@vercel
Copy link

vercel bot commented Feb 1, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Building Building Preview, Comment Feb 1, 2026 0:30am

Request Review

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@shin-bot-litellm
Copy link
Contributor Author

Closing in favor of PR from properly named branch to trigger CircleCI tests.

@shin-bot-litellm shin-bot-litellm deleted the fix/mapped-tests-core-scientific-notation-and-mocks branch February 1, 2026 00:32
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 1, 2026

Greptile Overview

Greptile Summary

Fixes test isolation and mock injection issues in four failing tests caused by LRU caching and HTTP client caching introduced in #19606.

Key Changes:

  • test_register_model_with_scientific_notation: Uses unique model name and properly clears LRU caches before/after test to prevent conflicts
  • test_search_uses_registry_credentials: Switches from string-based patching to patch.object() on the actual instance for more reliable mocking
  • test_send_email_missing_api_key: Directly injects mock HTTP client into logger instance to bypass caching issues
  • test_stream_transformation_error_sync: Simplifies mock path from module-level to direct litellm.completion patching

All changes are test-only with no production code impact. The fixes properly address cache-related test isolation issues.

Confidence Score: 5/5

  • This PR is safe to merge - test-only changes with proper isolation fixes
  • All changes are limited to test files, addressing specific test isolation and mocking issues. The fixes use established patterns (direct mock injection, unique test names, cache clearing). Only minor style improvements suggested.
  • No files require special attention

Important Files Changed

Filename Overview
tests/test_litellm/enterprise/enterprise_callbacks/send_emails/test_resend_email.py Improves mock injection by directly assigning mock client to logger instance, preventing cache-related issues
tests/test_litellm/google_genai/test_google_genai_handler.py Simplifies patch path from module-level to direct litellm.completion patching
tests/test_litellm/test_utils.py Adds proper test isolation with unique model name, cache clearing, and cleanup
tests/test_litellm/vector_stores/test_vector_store_registry.py Fixes instance method patching by using patch.object() on imported handler instance

Sequence Diagram

sequenceDiagram
    participant Test as Test Case
    participant Cache as LRU Cache/Client Cache
    participant Mock as Mock Objects
    participant SUT as System Under Test
    
    Note over Test,SUT: Before Fix: Cache Issues
    Test->>Cache: Initialize (cached data exists)
    Test->>Mock: Setup mock (ineffective)
    Cache-->>SUT: Returns cached real client
    SUT->>SUT: Uses real client (test fails)
    
    Note over Test,SUT: After Fix: Proper Isolation
    Test->>Cache: Clear caches (_invalidate_model_cost_lowercase_map)
    Test->>Test: Use unique identifiers (test-model-unique-12345)
    Test->>Mock: Create mock client
    Test->>SUT: Direct injection (logger.async_httpx_client = mock)
    Test->>Mock: Setup patch.object(instance, method)
    SUT->>Mock: Uses injected mock (test passes)
    Test->>Cache: Cleanup (delete entries, clear caches)
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

del litellm.model_cost[test_model_name]

# Clear LRU caches that might have stale data
from litellm.utils import get_model_info, _cached_get_model_info_helper, _invalidate_model_cost_lowercase_map
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_model_info and _cached_get_model_info_helper are imported but never used

Suggested change
from litellm.utils import get_model_info, _cached_get_model_info_helper, _invalidate_model_cost_lowercase_map
from litellm.utils import _invalidate_model_cost_lowercase_map
Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/test_litellm/test_utils.py
Line: 2293:2293

Comment:
`get_model_info` and `_cached_get_model_info_helper` are imported but never used

```suggestion
    from litellm.utils import _invalidate_model_cost_lowercase_map
```

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines +186 to +187
# Patch litellm.completion directly to prevent real API calls
with patch("litellm.completion", return_value=mock_stream):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inconsistent with the async test approach at line 215 which patches litellm.google_genai.adapters.handler.litellm. Consider using the same direct patching approach for consistency

Suggested change
# Patch litellm.completion directly to prevent real API calls
with patch("litellm.completion", return_value=mock_stream):
with patch("litellm.google_genai.adapters.handler.litellm") as mock_litellm:
mock_litellm.completion = MagicMock(return_value=mock_stream)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/test_litellm/google_genai/test_google_genai_handler.py
Line: 186:187

Comment:
inconsistent with the async test approach at line 215 which patches `litellm.google_genai.adapters.handler.litellm`. Consider using the same direct patching approach for consistency

```suggestion
        with patch("litellm.google_genai.adapters.handler.litellm") as mock_litellm:
            mock_litellm.completion = MagicMock(return_value=mock_stream)
```

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants