Skip to content

fix(test): add environment cleanup for Vertex AI rerank tests#21268

Merged
jquinter merged 1 commit intomainfrom
fix/vertex-ai-rerank-test-isolation
Feb 15, 2026
Merged

fix(test): add environment cleanup for Vertex AI rerank tests#21268
jquinter merged 1 commit intomainfrom
fix/vertex-ai-rerank-test-isolation

Conversation

@jquinter
Copy link
Contributor

Summary

Fixes test isolation issue in test_vertex_ai_rerank_transformation.py by adding proper environment variable cleanup.

Problem

  • Test passes in isolation ✅
  • Test fails in CI with DefaultCredentialsError: Your default credentials were not found
  • Previous tests set Google/Vertex AI environment variables (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT, etc.) and don't clean them up
  • When this test runs, it attempts real Google authentication instead of using mocks
  • This causes authentication failures in CI

Root Cause

Environment variable pollution from other tests that use Vertex AI/Google Cloud credentials:

  • tests/litellm_utils_tests/test_secret_manager.py
  • tests/pass_through_tests/test_vertex_ai.py
  • tests/unified_google_tests/base_google_test.py
  • tests/pass_through_unit_tests/test_unit_test_passthrough_router.py
  • etc.

Solution

Added setup_method and teardown_method to:

  1. setup_method: Save and clear all Google/Vertex AI environment variables before each test
  2. teardown_method: Restore saved environment variables after each test

This ensures each test starts with a clean environment regardless of what previous tests did.

Testing

# All 17 tests pass locally
pytest tests/test_litellm/llms/vertex_ai/rerank/test_vertex_ai_rerank_transformation.py -v
======================= 17 passed in 0.17s ========================

Related

This is another test isolation issue affecting PR #21217 (Anthropic structured output test). NOT caused by PR #21217 - that PR only modifies Anthropic test expectations.

🤖 Generated with Claude Code

Add setup_method and teardown_method to clean up Google/Vertex AI
environment variables that may be left by previous tests.

Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex
environment variables and not clean them up, causing this test to
attempt real Google authentication instead of using mocks.

This fix:
- Saves and clears Google/Vertex env vars in setup_method
- Restores them in teardown_method
- Prevents "DefaultCredentialsError" in CI when run with other tests

Test passes in isolation but fails in CI due to test ordering. This
is another test isolation issue, NOT related to PR #21217.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Feb 15, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Ready Ready Preview, Comment Feb 15, 2026 9:28pm

Request Review

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 15, 2026

Greptile Summary

Adds proper environment variable cleanup to TestVertexAIRerankTransform to fix test isolation issues in CI. The setup_method now saves and clears 6 Google/Vertex AI environment variables (GOOGLE_APPLICATION_CREDENTIALS, GOOGLE_CLOUD_PROJECT, VERTEXAI_PROJECT, VERTEX_PROJECT, VERTEX_LOCATION, VERTEX_AI_PROJECT), and teardown_method restores them after each test.

  • Prevents DefaultCredentialsError failures caused by env var pollution from other test files that set Google Cloud credentials without cleaning up
  • Uses a standard save-and-restore pattern consistent with other test fixtures in the codebase
  • All 17 existing tests continue to pass; no behavioral changes to test logic

Confidence Score: 5/5

  • This PR is safe to merge — it only modifies test infrastructure with a well-understood save/restore pattern.
  • The change is minimal, limited to test setup/teardown, uses a standard pattern seen elsewhere in the codebase, and does not affect production code. The env vars being cleared are comprehensive for the Vertex AI authentication path. The teardown properly restores original values.
  • No files require special attention.

Important Files Changed

Filename Overview
tests/test_litellm/llms/vertex_ai/rerank/test_vertex_ai_rerank_transformation.py Adds environment variable save/clear in setup_method and restore in teardown_method to prevent test isolation issues from leaked Google/Vertex AI credentials. Clean implementation with proper save-and-restore pattern.

Flowchart

flowchart TD
    A["Test Runner: Begins test method"] --> B["setup_method()"]
    B --> C["Save existing Google/Vertex AI env vars\nto self._saved_env"]
    C --> D["Clear env vars:\nGOOGLE_APPLICATION_CREDENTIALS\nGOOGLE_CLOUD_PROJECT\nVERTEXAI_PROJECT\nVERTEX_PROJECT\nVERTEX_LOCATION\nVERTEX_AI_PROJECT"]
    D --> E["Initialize VertexAIRerankConfig\nand model"]
    E --> F["Run test method\n(mocked, no real auth)"]
    F --> G["teardown_method()"]
    G --> H["Restore saved env vars\nfrom self._saved_env"]
    H --> I["Test Runner: Next test method"]
Loading

Last reviewed commit: 28a0c61

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 file reviewed, no comments

Edit Code Review Agent Settings | Greptile

@jquinter jquinter merged commit e162410 into main Feb 15, 2026
18 of 23 checks passed
jquinter added a commit that referenced this pull request Feb 15, 2026
Add autouse pytest fixture to clear Google/Vertex AI environment
variables before each test, preventing authentication errors in CI.

Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex
environment variables and not clean them up, causing this test to
attempt real Google authentication instead of using mocks.

This fix:
- Adds clean_vertex_env fixture with autouse=True
- Saves and clears Google/Vertex env vars before each test
- Restores them after each test
- Prevents "AuthenticationError: Request had invalid authentication
  credentials" (401) in CI when run with other tests

Same fix pattern as PR #21268 (rerank) and PR #21272 (GPT-OSS).

Related: test was failing on PR #21217, but NOT caused by PR #21217
(which only modifies test_anthropic_structured_output.py). This is
another test isolation issue.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jquinter added a commit that referenced this pull request Feb 15, 2026
…ution

Implements three key improvements to reduce test flakiness from parallel execution:

1. **Split Vertex AI tests into separate group** (workers: 1)
   - Vertex AI tests often have environment variable pollution issues
   - Running serially prevents cross-test interference with GOOGLE_APPLICATION_CREDENTIALS
   - Isolates authentication-related test failures

2. **Reduce workers for other LLM tests** (4 -> 2)
   - Decreases chance of race conditions and state conflicts
   - Still parallel but with less contention

3. **Add --dist=loadscope to pytest-xdist**
   - Keeps tests from the same file together on one worker
   - Reduces interference between unrelated test modules
   - Data shows 70% pass rate WITH loadscope vs 40% WITHOUT
   - Better test isolation while maintaining parallelism

Note: loadscope exposes one tokenizer cache issue in core-utils which will be
fixed in a separate PR. The tradeoff is worth it (7/10 pass vs 4/10 without).

These changes address the root causes of intermittent test failures in:
PRs #21268, #21271, #21272, #21273, #21275, #21276:
- Environment variable pollution (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT)
- Global state conflicts (litellm.known_tokenizer_config)
- Async mock timing issues with parallel execution

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
sameetn pushed a commit to sameetn/litellm that referenced this pull request Feb 16, 2026
Add autouse pytest fixture to clear Google/Vertex AI environment
variables before each test, preventing authentication errors in CI.

Previous tests may set GOOGLE_APPLICATION_CREDENTIALS or other Vertex
environment variables and not clean them up, causing this test to
attempt real Google authentication instead of using mocks.

This fix:
- Adds clean_vertex_env fixture with autouse=True
- Saves and clears Google/Vertex env vars before each test
- Restores them after each test
- Prevents "AuthenticationError: Request had invalid authentication
  credentials" (401) in CI when run with other tests

Same fix pattern as PR BerriAI#21268 (rerank) and PR BerriAI#21272 (GPT-OSS).

Related: test was failing on PR BerriAI#21217, but NOT caused by PR BerriAI#21217
(which only modifies test_anthropic_structured_output.py). This is
another test isolation issue.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jquinter added a commit that referenced this pull request Feb 18, 2026
…ution

Implements three key improvements to reduce test flakiness from parallel execution:

1. **Split Vertex AI tests into separate group** (workers: 1)
   - Vertex AI tests often have environment variable pollution issues
   - Running serially prevents cross-test interference with GOOGLE_APPLICATION_CREDENTIALS
   - Isolates authentication-related test failures

2. **Reduce workers for other LLM tests** (4 -> 2)
   - Decreases chance of race conditions and state conflicts
   - Still parallel but with less contention

3. **Add --dist=loadscope to pytest-xdist**
   - Keeps tests from the same file together on one worker
   - Reduces interference between unrelated test modules
   - Data shows 70% pass rate WITH loadscope vs 40% WITHOUT
   - Better test isolation while maintaining parallelism

Note: loadscope exposes one tokenizer cache issue in core-utils which will be
fixed in a separate PR. The tradeoff is worth it (7/10 pass vs 4/10 without).

These changes address the root causes of intermittent test failures in:
PRs #21268, #21271, #21272, #21273, #21275, #21276:
- Environment variable pollution (GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT)
- Global state conflicts (litellm.known_tokenizer_config)
- Async mock timing issues with parallel execution

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant