improve(ci): enhance test stability with better isolation and distribution

[jquinter]

Summary

Implements three key improvements to reduce test flakiness caused by parallel execution in CI.

Problem

Tests have been failing intermittently in CI due to:

• Parallel execution with pytest-xdist causing race conditions
• Environment variable pollution across tests
• Global state conflicts (e.g., litellm.known_tokenizer_config)
• Tests passing in isolation but failing in CI

Changes

1. Split Vertex AI Tests into Separate Group ✨

- name: "llms-vertex"
  path: "tests/test_litellm/llms/vertex_ai"
  workers: 1  # Serial execution

Why: Vertex AI tests are particularly sensitive to:

• GOOGLE_APPLICATION_CREDENTIALS pollution
• vertexai module import triggering authentication
• Environment variable conflicts

Benefit: Complete isolation prevents auth-related failures

2. Reduce Workers for Other LLM Tests 🔧

- name: "llms-other"
  path: "tests/test_litellm/llms --ignore=tests/test_litellm/llms/vertex_ai"
  workers: 2  # Reduced from 4

Why: Less parallelism = fewer race conditions
Benefit: Still parallel but with reduced contention

3. Add --dist=loadscope to pytest-xdist 🎯

--dist=loadscope

Why: Keeps tests from the same file together on one worker
Benefit: Reduces cross-module interference while maintaining parallelism

Related PRs

This addresses the root causes behind recent test stability fixes:

• fix(test): add environment cleanup for Vertex AI rerank tests #21268 - Vertex AI rerank environment cleanup
• fix(test): update reasoning_effort test to expect dict format #21271 - Reasoning effort test expectations
• fix(test): add environment cleanup for Vertex AI GPT-OSS tests #21272 - Vertex AI GPT-OSS environment cleanup
• fix(test): add environment cleanup for Vertex AI Qwen tests #21273 - Vertex AI Qwen environment cleanup
• fix(test): use async side_effect for client.post mock in watsonx test #21275 - WatsonX async mock fix
• fix(test): mock vertexai module in GPT-OSS tests to prevent authentication #21276 - Vertex AI GPT-OSS vertexai module mock

Testing

The workflow syntax is valid and will be tested on the next CI run.

Impact

• Slightly longer CI runtime for Vertex AI tests (serial vs parallel)
• Much higher reliability - fewer intermittent failures
• Better debuggability - clearer test isolation

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 18, 2026 0:51am

[jquinter]

@greptile-apps review please

[jquinter]

Update: Removed --dist=loadscope

Issue Found:
The --dist=loadscope change exposed a pre-existing test isolation issue in test_token_counter.py. The tokenizer cache persists between tests when they run on the same worker, causing:

AssertionError: Expected 'from_pretrained' to be called once. Called 0 times.

Root Cause:
When tests from the same file run together (loadscope behavior), the tokenizer gets cached from earlier tests, so later tests never call from_pretrained.

Solution:
Removed --dist=loadscope from this PR. The test isolation issue should be fixed separately by either:

1. Adding proper cache cleanup between tests
2. Using @pytest.fixture(autouse=True) to clear tokenizer caches

What Remains:
Still keeping the two valuable improvements that don't cause issues:
✅ Vertex AI tests isolated (workers: 1)
✅ LLM tests reduced parallelism (workers: 2)

These alone should significantly improve test stability without the side effects.

[jquinter]

@greptile-apps please re-review

[greptile-apps]

Greptile Summary

This PR improves CI test stability by making three changes to the test matrix workflow:

• Splits Vertex AI tests into a dedicated llms-vertex group running serially (workers: 1) to avoid auth/env variable pollution that caused intermittent failures
• Isolates remaining LLM tests into llms-other group with --ignore to exclude Vertex AI, keeping 2 workers
• Adds --dist=loadscope to all pytest-xdist runs so tests from the same file stay on the same worker, reducing cross-module interference

The changes are straightforward CI configuration adjustments with a clear tradeoff: slightly longer Vertex AI test runtime in exchange for significantly fewer intermittent failures. This complements a series of related PRs (#21268, #21271, #21272, #21273, #21275, #21276) that addressed individual test isolation issues.

Confidence Score: 4/5

• This PR is safe to merge — it only modifies CI workflow configuration and does not touch application or test code.
• Score of 4 reflects that the changes are limited to CI workflow YAML with no application code impact. The --ignore syntax and --dist=loadscope are well-established pytest-xdist features. The only minor concern is that --dist=loadscope with -n 1 is technically redundant (harmless but unnecessary), and the effectiveness of these changes can only be validated by observing subsequent CI runs.
• No files require special attention — the only changed file is a CI workflow configuration.

Important Files Changed

Filename	Overview
.github/workflows/test-litellm-matrix.yml	Splits Vertex AI tests into a separate serial group, reduces parallelism for other LLM tests, and adds --dist=loadscope globally. Changes are low-risk CI configuration improvements targeting test flakiness.

Flowchart

flowchart TD
    A["test-litellm-matrix.yml"] --> B{"Matrix Strategy"}
    B --> C["llms-vertex\n(NEW - workers: 1, serial)"]
    B --> D["llms-other\n(NEW - workers: 2, --ignore vertex_ai)"]
    B --> E["proxy-guardrails\n(workers: 2)"]
    B --> F["proxy-core\n(workers: 2)"]
    B --> G["proxy-misc\n(workers: 2)"]
    B --> H["integrations\n(workers: 2)"]
    B --> I["core-utils\n(workers: 2)"]
    B --> J["other\n(workers: 2)"]
    B --> K["root\n(workers: 2)"]
    B --> L["proxy-unit-a\n(workers: 2)"]
    B --> M["proxy-unit-b\n(workers: 2)"]

    C --> N["--dist=loadscope (NEW)\n+ --reruns 2"]
    D --> N
    E --> N
    F --> N
    G --> N
    H --> N
    I --> N
    J --> N
    K --> N
    L --> N
    M --> N

    style C fill:#90EE90,stroke:#333
    style D fill:#90EE90,stroke:#333
    style N fill:#87CEEB,stroke:#333

Loading

_{Last reviewed commit: 44feb55}

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

--dist=loadscope is a no-op with -n 1

For the llms-vertex group (workers: 1), --dist=loadscope has no effect since there's only one worker — there's nothing to distribute. This is harmless, but you could consider conditionally applying it only when workers > 1 for clarity, or just leave it as-is for simplicity since the overhead is negligible.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}