fix(tests): resolve test isolation issue in http_handler tests#21388
Merged
fix(tests): resolve test isolation issue in http_handler tests#21388
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryThis PR fixes intermittent CI test failures in
Confidence Score: 4/5
|
| Filename | Overview |
|---|---|
| tests/test_litellm/llms/custom_httpx/test_http_handler.py | Moves AsyncHTTPHandler imports into test function scope (aliased as AsyncHTTPHandlerReload) for 3 tests that use isinstance() checks, preventing stale class references after importlib.reload(litellm) in conftest.py. Also includes minor whitespace normalization (trailing spaces removed). The fix is correctly scoped to only the affected tests. |
Sequence Diagram
sequenceDiagram
participant C as conftest.py (module scope)
participant T as test_http_handler.py
participant L as litellm module
participant H as http_handler module
Note over T: Module-level import at load time
T->>H: import AsyncHTTPHandler (OLD ref)
Note over C: Fixture runs before each test module
C->>L: importlib.reload(litellm)
L->>H: Re-imports http_handler (NEW class)
Note over T: BEFORE fix (fails)
T->>H: get_async_httpx_client() returns NEW instance
T--xT: isinstance(obj, OLD AsyncHTTPHandler) → False ❌
Note over T: AFTER fix (passes)
T->>H: from http_handler import AsyncHTTPHandler (NEW ref)
T->>H: get_async_httpx_client() returns NEW instance
T->>T: isinstance(obj, NEW AsyncHTTPHandler) → True ✅
Last reviewed commit: 821ed87
jquinter
added a commit
that referenced
this pull request
Feb 17, 2026
Add two detailed guides for addressing CI test flakiness: 1. test-flakiness-guide.md - Developer guide with: - How to use @pytest.mark.no_parallel for async mocks - Patterns for robust async mock setup - Retry logic strategies - Module reload issues and fixes - Quick reference and checklist 2. ci-test-improvements.md - Implementation plan with: - Priority phased rollout (Quick wins → CI → Enforcement) - pytest-rerunfailures plugin setup - GitHub Actions improvements for retries - Makefile targets for testing - Pre-commit hooks for test quality - Test utilities module with decorators - Success metrics and monitoring These guides provide actionable solutions for the CI test failures observed in PRs #21107, #21388, and #21390. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
821ed87 to
a4d3613
Compare
8b8d520 to
a4d3613
Compare
a4d3613 to
978a59d
Compare
Fix isinstance() checks failing due to module reload in conftest.py. The conftest.py fixture reloads the litellm module between test modules, which causes class references imported at module-level to become stale. When AsyncHTTPHandler is imported at the top of the file and then litellm is reloaded by the fixture, the isinstance() check fails because the returned instance is of the NEW AsyncHTTPHandler class while the test is checking against the OLD class reference. Solution: Import AsyncHTTPHandler locally within each test function that uses isinstance() checks. This ensures we get the fresh class reference after the module reload. Fixed tests: - test_session_reuse_integration - test_get_async_httpx_client_with_shared_session - test_get_async_httpx_client_without_shared_session This resolves intermittent CI failures where parallel test execution triggers the module reload behavior. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
After rebasing with main, pyproject.toml contains dependency changes from PR #21394 (removed pytest-retry, added pytest-xdist). Running `poetry lock` to sync the lock file with the updated pyproject.toml. This resolves the CI error: 'pyproject.toml changed significantly since poetry.lock was last generated'
1ee3ad8 to
fb839d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Tests in
test_http_handler.pywere failing intermittently in CI with:Root Cause
The
conftest.pyfixture reloads thelitellmmodule between test modules withimportlib.reload(litellm). This causes class references imported at module-level to become stale:AsyncHTTPHandlerat the top (module-level)AsyncHTTPHandlerclassget_async_httpx_client()- gets instance of NEW classisinstance(obj, AsyncHTTPHandler)- compares against OLD class referenceSolution
Import
AsyncHTTPHandlerlocally within each test function that usesisinstance()checks. This ensures we get the fresh class reference after the module reload.Fixed Tests
test_session_reuse_integrationtest_get_async_httpx_client_with_shared_sessiontest_get_async_httpx_client_without_shared_sessionImpact
Related
This issue was discovered while investigating test failures in PR #21107 (unrelated to that PR's changes).