Fix test isolation for test_log_langfuse_v2_handles_null_usage_values#20475
Conversation
Create fresh mock objects within the test instead of reusing mocks from setUp that have side_effect configured. The setUp's side_effect on mock_langfuse_client.trace can interfere with return_value settings when tests try to reset and reconfigure mocks. Using dedicated mock objects for this test avoids state pollution from setUp's side_effect configuration and makes the test more deterministic in parallel execution environments.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Greptile OverviewGreptile SummaryThis PR fixes a test isolation issue in Root cause: The test's Solution: Create fresh Changes:
Confidence Score: 5/5
|
| Filename | Overview |
|---|---|
| tests/test_litellm/integrations/test_langfuse.py | Replaced shared mocks with fresh test-scoped mocks to fix race condition caused by side_effect precedence over return_value |
Sequence Diagram
sequenceDiagram
participant Test as test_log_langfuse_v2_handles_null_usage_values
participant Logger as LangFuseLogger
participant MockClient as mock_client
participant MockTrace as mock_trace
participant MockGen as mock_generation
Note over Test: OLD: Used setUp's mocks with side_effect
Note over Test: NEW: Creates fresh mocks
Test->>Test: Create mock_client, mock_trace, mock_generation
Test->>Test: Configure chain: client.trace() → trace
Test->>Test: Configure chain: trace.generation() → generation
Test->>Logger: Assign mock_client to logger.Langfuse
Test->>Logger: Call _log_langfuse_v2(response_obj with null usage)
Logger->>MockClient: trace(name, metadata, ...)
MockClient-->>Logger: Returns mock_trace
Logger->>MockTrace: generation(usage={prompt_tokens:0, ...})
MockTrace-->>Logger: Returns mock_generation
Test->>MockClient: assert trace.assert_called()
Test->>MockTrace: assert generation.assert_called_once()
Test->>Test: Verify usage values converted None→0
Fixes test_log_langfuse_v2_handles_null_usage_values flaky test failure by properly cleaning up sys.modules['langfuse'] in tearDown. Changes: - Store original langfuse module in setUp before mocking - Restore original or remove mock in tearDown to prevent state pollution - Remove invalid print_verbose parameter from log_event_on_langfuse Root Cause: The tearDown method was not cleaning up sys.modules['langfuse'] after each test, causing mock state to leak between tests. This caused intermittent failures in CI, especially when tests run in parallel or in different orders. Impact: This test has a long history of flakiness with multiple attempted fixes (#20475, #17599, #17594, #17591, #17588). The missing sys.modules cleanup was the underlying issue causing continued failures despite those patches. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixes test_log_langfuse_v2_handles_null_usage_values flaky test failure by properly cleaning up sys.modules['langfuse'] in tearDown. Changes: - Store original langfuse module in setUp before mocking - Restore original or remove mock in tearDown to prevent state pollution - Remove invalid print_verbose parameter from log_event_on_langfuse Root Cause: The tearDown method was not cleaning up sys.modules['langfuse'] after each test, causing mock state to leak between tests. This caused intermittent failures in CI, especially when tests run in parallel or in different orders. Impact: This test has a long history of flakiness with multiple attempted fixes (#20475, #17599, #17594, #17591, #17588). The missing sys.modules cleanup was the underlying issue causing continued failures despite those patches. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixes test_log_langfuse_v2_handles_null_usage_values flaky test failure by properly cleaning up sys.modules['langfuse'] in tearDown. Changes: - Store original langfuse module in setUp before mocking - Restore original or remove mock in tearDown to prevent state pollution - Remove invalid print_verbose parameter from log_event_on_langfuse Root Cause: The tearDown method was not cleaning up sys.modules['langfuse'] after each test, causing mock state to leak between tests. This caused intermittent failures in CI, especially when tests run in parallel or in different orders. Impact: This test has a long history of flakiness with multiple attempted fixes (BerriAI#20475, BerriAI#17599, BerriAI#17594, BerriAI#17591, BerriAI#17588). The missing sys.modules cleanup was the underlying issue causing continued failures despite those patches. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Regression Fix
Failing Job:
litellm_mapped_tests_integrationsCaused By: Test isolation issue introduced in 1017c3a
Author: @ishaan-jaff
What Broke
Test
test_log_langfuse_v2_handles_null_usage_valuesfails intermittently when running with parallel test execution.Error:
Original Commit LOC
The test originally:
reset_mock()on setUp's mocksreturn_valueon mocks that hadside_effectconfiguredside_effecttakes precedence overreturn_value, causing mock chain issuesThis Fix
Create fresh mock objects within the test instead of reusing mocks from setUp:
side_effectconfigurationThis ensures the test is deterministic in parallel execution environments.