litellm_fix_mapped_tests_core: clear client cache and fix isinstance checks#20195
litellm_fix_mapped_tests_core: clear client cache and fix isinstance checks#20195shin-bot-litellm wants to merge 1 commit intomainfrom
Conversation
…checks ## Problem Tests using mocked HTTP clients were hitting real APIs because: 1. HTTP client cache was returning previously cached real clients 2. isinstance checks failed due to module identity issues from sys.path ### Tests affected: - test_send_email_missing_api_key - test_send_email_multiple_recipients (resend & sendgrid) - test_search_uses_registry_credentials - test_vector_store_create_with_simple_provider_name - test_image_edit_merges_headers_and_extra_headers ## Solution 1. Add clear_client_cache fixture (autouse=True) to clear litellm.in_memory_llm_clients_cache before each test 2. Fix isinstance checks to use type name comparison (avoids module identity issues from sys.path.insert) ## Why not disable_aiohttp_transport The default transport is aiohttp, so tests should work with it. Clearing the cache ensures mocks are used instead of cached real clients. ## Regression PR #19829 (commit f95572e) added @respx.mock but cached clients from earlier tests were being reused, bypassing the mocks.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Greptile OverviewGreptile SummaryThis PR fixes test isolation issues where cached HTTP clients bypassed mocks, causing tests to make real API calls instead of using mocked responses. Changes:
Root Cause: Impact:
Confidence Score: 5/5
|
| Filename | Overview |
|---|---|
| tests/test_litellm/enterprise/enterprise_callbacks/send_emails/test_resend_email.py | Added clear_client_cache fixture to flush HTTP client cache before/after each test, preventing cached real clients from bypassing mocks |
| tests/test_litellm/enterprise/enterprise_callbacks/send_emails/test_sendgrid_email.py | Added clear_client_cache fixture to flush HTTP client cache before/after each test, ensuring test isolation with mocked clients |
| tests/test_litellm/test_main.py | Added clear_client_cache fixture to flush HTTP client cache before/after each test, ensuring mocks are used instead of cached clients |
| tests/test_litellm/vector_stores/test_vector_store_create_provider_logic.py | Replaced isinstance checks with type name comparisons to avoid module identity issues caused by sys.path manipulation in test setup |
| tests/test_litellm/vector_stores/test_vector_store_registry.py | Added clear_client_cache fixture to flush HTTP client cache before/after each test, preventing real API calls when mocks should be used |
Sequence Diagram
sequenceDiagram
participant Test as Test Function
participant Fixture as clear_client_cache Fixture
participant Cache as in_memory_llm_clients_cache
participant HTTPHandler as get_async_httpx_client()
participant Mock as @respx.mock / Mock Client
Note over Fixture,Cache: Before Test Execution
Fixture->>Cache: flush_cache()
Cache-->>Fixture: Cache cleared
Note over Test,Mock: Test Execution
Test->>HTTPHandler: Request HTTP client
HTTPHandler->>Cache: get_cache(key)
Cache-->>HTTPHandler: None (no cached client)
HTTPHandler->>Mock: Create new client (uses mock)
Mock-->>HTTPHandler: Mocked client
HTTPHandler->>Cache: set_cache(key, client)
HTTPHandler-->>Test: Return mocked client
Test->>Mock: Make HTTP request
Mock-->>Test: Mocked response (no real API call)
Note over Fixture,Cache: After Test Execution
Fixture->>Cache: flush_cache()
Cache-->>Fixture: Cache cleared
Problem
Tests using mocked HTTP clients were hitting real APIs because:
isinstancechecks failed due to module identity issues fromsys.pathTests affected:
test_send_email_missing_api_keytest_send_email_multiple_recipients(resend & sendgrid)test_search_uses_registry_credentialstest_vector_store_create_with_simple_provider_nametest_image_edit_merges_headers_and_extra_headersSolution
clear_client_cachefixture (autouse=True) to clearlitellm.in_memory_llm_clients_cachebefore each testisinstancechecks to use type name comparison (avoids module identity issues fromsys.path.insert)Why not disable_aiohttp_transport
The default transport is aiohttp, so tests should work with it. Clearing the cache ensures mocks are used instead of cached real clients.
Regression
PR #19829 (commit
f95572e3ed) added@respx.mockbut cached clients from earlier tests were being reused, bypassing the mocks.