fix(test): add environment cleanup for Vertex AI Qwen tests#21273
Merged
fix(test): add environment cleanup for Vertex AI Qwen tests#21273
Conversation
Add autouse pytest fixture to clear Google/Vertex AI environment variables before each test, preventing authentication errors in CI. Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex environment variables and not clean them up, causing this test to attempt real Google authentication instead of using mocks. This fix: - Adds clean_vertex_env fixture with autouse=True - Saves and clears Google/Vertex env vars before each test - Restores them after each test - Prevents "AuthenticationError: Request had invalid authentication credentials" (401) in CI when run with other tests Same fix pattern as PR #21268 (rerank) and PR #21272 (GPT-OSS). Related: test was failing on PR #21217, but NOT caused by PR #21217 (which only modifies test_anthropic_structured_output.py). This is another test isolation issue. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryThis PR adds a
Confidence Score: 5/5
|
| Filename | Overview |
|---|---|
| tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/qwen/test_vertex_ai_qwen_global_endpoint.py | Adds autouse pytest fixture to clean Google/Vertex AI environment variables before each test and restore them after. Standard test isolation pattern consistent with sibling PRs. |
Sequence Diagram
sequenceDiagram
participant Pytest as Pytest Runner
participant Fixture as clean_vertex_env Fixture
participant Env as os.environ
participant Test as Test Function
Pytest->>Fixture: Before each test (autouse)
Fixture->>Env: Save & clear GOOGLE_APPLICATION_CREDENTIALS
Fixture->>Env: Save & clear GOOGLE_CLOUD_PROJECT
Fixture->>Env: Save & clear VERTEXAI_PROJECT
Fixture->>Env: Save & clear VERTEX_PROJECT
Fixture->>Env: Save & clear VERTEX_LOCATION
Fixture->>Env: Save & clear VERTEX_AI_PROJECT
Fixture-->>Pytest: yield (env is clean)
Pytest->>Test: Run test with clean environment
Test-->>Pytest: Test completes
Pytest->>Fixture: After test (teardown)
Fixture->>Env: Restore saved variables
Fixture-->>Pytest: Cleanup done
Last reviewed commit: 62ac8ce
jquinter
added a commit
that referenced
this pull request
Feb 15, 2026
…ution Implements three key improvements to reduce test flakiness from parallel execution: 1. **Split Vertex AI tests into separate group** (workers: 1) - Vertex AI tests often have environment variable pollution issues - Running serially prevents cross-test interference with GOOGLE_APPLICATION_CREDENTIALS - Isolates authentication-related test failures 2. **Reduce workers for other LLM tests** (4 -> 2) - Decreases chance of race conditions and state conflicts - Still parallel but with less contention 3. **Add --dist=loadscope to pytest-xdist** - Keeps tests from the same file together on one worker - Reduces interference between unrelated test modules - Data shows 70% pass rate WITH loadscope vs 40% WITHOUT - Better test isolation while maintaining parallelism Note: loadscope exposes one tokenizer cache issue in core-utils which will be fixed in a separate PR. The tradeoff is worth it (7/10 pass vs 4/10 without). These changes address the root causes of intermittent test failures in: PRs #21268, #21271, #21272, #21273, #21275, #21276: - Environment variable pollution (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT) - Global state conflicts (litellm.known_tokenizer_config) - Async mock timing issues with parallel execution Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jquinter
added a commit
that referenced
this pull request
Feb 18, 2026
…ution Implements three key improvements to reduce test flakiness from parallel execution: 1. **Split Vertex AI tests into separate group** (workers: 1) - Vertex AI tests often have environment variable pollution issues - Running serially prevents cross-test interference with GOOGLE_APPLICATION_CREDENTIALS - Isolates authentication-related test failures 2. **Reduce workers for other LLM tests** (4 -> 2) - Decreases chance of race conditions and state conflicts - Still parallel but with less contention 3. **Add --dist=loadscope to pytest-xdist** - Keeps tests from the same file together on one worker - Reduces interference between unrelated test modules - Data shows 70% pass rate WITH loadscope vs 40% WITHOUT - Better test isolation while maintaining parallelism Note: loadscope exposes one tokenizer cache issue in core-utils which will be fixed in a separate PR. The tradeoff is worth it (7/10 pass vs 4/10 without). These changes address the root causes of intermittent test failures in: PRs #21268, #21271, #21272, #21273, #21275, #21276: - Environment variable pollution (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT) - Global state conflicts (litellm.known_tokenizer_config) - Async mock timing issues with parallel execution Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes test isolation issue in Vertex AI Qwen tests by adding environment variable cleanup.
Problem
Test
test_vertex_ai_qwen_global_endpoint_urlwas failing in CI with:Previous tests set Google/Vertex AI environment variables and don't clean them up, causing this test to attempt real Google authentication instead of using mocks.
Solution
Added
clean_vertex_envpytest fixture withautouse=Trueto:Same fix pattern as:
Testing
Prevents real API calls by ensuring clean environment for all tests in this file.
Related
This test failure was reported on PR #21217, but NOT caused by PR #21217 (which only modifies Anthropic structured output tests). This is part of a broader test isolation issue affecting multiple Vertex AI test files.
🤖 Generated with Claude Code