fix(test): add environment cleanup for Vertex AI rerank tests#21268
Merged
fix(test): add environment cleanup for Vertex AI rerank tests#21268
Conversation
Add setup_method and teardown_method to clean up Google/Vertex AI environment variables that may be left by previous tests. Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex environment variables and not clean them up, causing this test to attempt real Google authentication instead of using mocks. This fix: - Saves and clears Google/Vertex env vars in setup_method - Restores them in teardown_method - Prevents "DefaultCredentialsError" in CI when run with other tests Test passes in isolation but fails in CI due to test ordering. This is another test isolation issue, NOT related to PR #21217. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryAdds proper environment variable cleanup to
Confidence Score: 5/5
|
| Filename | Overview |
|---|---|
| tests/test_litellm/llms/vertex_ai/rerank/test_vertex_ai_rerank_transformation.py | Adds environment variable save/clear in setup_method and restore in teardown_method to prevent test isolation issues from leaked Google/Vertex AI credentials. Clean implementation with proper save-and-restore pattern. |
Flowchart
flowchart TD
A["Test Runner: Begins test method"] --> B["setup_method()"]
B --> C["Save existing Google/Vertex AI env vars\nto self._saved_env"]
C --> D["Clear env vars:\nGOOGLE_APPLICATION_CREDENTIALS\nGOOGLE_CLOUD_PROJECT\nVERTEXAI_PROJECT\nVERTEX_PROJECT\nVERTEX_LOCATION\nVERTEX_AI_PROJECT"]
D --> E["Initialize VertexAIRerankConfig\nand model"]
E --> F["Run test method\n(mocked, no real auth)"]
F --> G["teardown_method()"]
G --> H["Restore saved env vars\nfrom self._saved_env"]
H --> I["Test Runner: Next test method"]
Last reviewed commit: 28a0c61
jquinter
added a commit
that referenced
this pull request
Feb 15, 2026
Add autouse pytest fixture to clear Google/Vertex AI environment variables before each test, preventing authentication errors in CI. Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex environment variables and not clean them up, causing this test to attempt real Google authentication instead of using mocks. This fix: - Adds clean_vertex_env fixture with autouse=True - Saves and clears Google/Vertex env vars before each test - Restores them after each test - Prevents "AuthenticationError: Request had invalid authentication credentials" (401) in CI when run with other tests Same fix pattern as PR #21268 (rerank) and PR #21272 (GPT-OSS). Related: test was failing on PR #21217, but NOT caused by PR #21217 (which only modifies test_anthropic_structured_output.py). This is another test isolation issue. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This was referenced Feb 15, 2026
jquinter
added a commit
that referenced
this pull request
Feb 15, 2026
…ution Implements three key improvements to reduce test flakiness from parallel execution: 1. **Split Vertex AI tests into separate group** (workers: 1) - Vertex AI tests often have environment variable pollution issues - Running serially prevents cross-test interference with GOOGLE_APPLICATION_CREDENTIALS - Isolates authentication-related test failures 2. **Reduce workers for other LLM tests** (4 -> 2) - Decreases chance of race conditions and state conflicts - Still parallel but with less contention 3. **Add --dist=loadscope to pytest-xdist** - Keeps tests from the same file together on one worker - Reduces interference between unrelated test modules - Data shows 70% pass rate WITH loadscope vs 40% WITHOUT - Better test isolation while maintaining parallelism Note: loadscope exposes one tokenizer cache issue in core-utils which will be fixed in a separate PR. The tradeoff is worth it (7/10 pass vs 4/10 without). These changes address the root causes of intermittent test failures in: PRs #21268, #21271, #21272, #21273, #21275, #21276: - Environment variable pollution (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT) - Global state conflicts (litellm.known_tokenizer_config) - Async mock timing issues with parallel execution Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
sameetn
pushed a commit
to sameetn/litellm
that referenced
this pull request
Feb 16, 2026
Add autouse pytest fixture to clear Google/Vertex AI environment variables before each test, preventing authentication errors in CI. Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex environment variables and not clean them up, causing this test to attempt real Google authentication instead of using mocks. This fix: - Adds clean_vertex_env fixture with autouse=True - Saves and clears Google/Vertex env vars before each test - Restores them after each test - Prevents "AuthenticationError: Request had invalid authentication credentials" (401) in CI when run with other tests Same fix pattern as PR BerriAI#21268 (rerank) and PR BerriAI#21272 (GPT-OSS). Related: test was failing on PR BerriAI#21217, but NOT caused by PR BerriAI#21217 (which only modifies test_anthropic_structured_output.py). This is another test isolation issue. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jquinter
added a commit
that referenced
this pull request
Feb 18, 2026
…ution Implements three key improvements to reduce test flakiness from parallel execution: 1. **Split Vertex AI tests into separate group** (workers: 1) - Vertex AI tests often have environment variable pollution issues - Running serially prevents cross-test interference with GOOGLE_APPLICATION_CREDENTIALS - Isolates authentication-related test failures 2. **Reduce workers for other LLM tests** (4 -> 2) - Decreases chance of race conditions and state conflicts - Still parallel but with less contention 3. **Add --dist=loadscope to pytest-xdist** - Keeps tests from the same file together on one worker - Reduces interference between unrelated test modules - Data shows 70% pass rate WITH loadscope vs 40% WITHOUT - Better test isolation while maintaining parallelism Note: loadscope exposes one tokenizer cache issue in core-utils which will be fixed in a separate PR. The tradeoff is worth it (7/10 pass vs 4/10 without). These changes address the root causes of intermittent test failures in: PRs #21268, #21271, #21272, #21273, #21275, #21276: - Environment variable pollution (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT) - Global state conflicts (litellm.known_tokenizer_config) - Async mock timing issues with parallel execution Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes test isolation issue in
test_vertex_ai_rerank_transformation.pyby adding proper environment variable cleanup.Problem
DefaultCredentialsError: Your default credentials were not found❌GOOGLE_APPLICATION_CREDENTIALS,VERTEXAI_PROJECT, etc.) and don't clean them upRoot Cause
Environment variable pollution from other tests that use Vertex AI/Google Cloud credentials:
tests/litellm_utils_tests/test_secret_manager.pytests/pass_through_tests/test_vertex_ai.pytests/unified_google_tests/base_google_test.pytests/pass_through_unit_tests/test_unit_test_passthrough_router.pySolution
Added
setup_methodandteardown_methodto:This ensures each test starts with a clean environment regardless of what previous tests did.
Testing
Related
This is another test isolation issue affecting PR #21217 (Anthropic structured output test). NOT caused by PR #21217 - that PR only modifies Anthropic test expectations.
🤖 Generated with Claude Code