[Infra] Optimize CI Pipeline

[yuneng-jiang]

Relevant issues

CI/CD now runs in ~11 mins

https://app.circleci.com/pipelines/github/BerriAI/litellm?branch=litellm_ci_optimize

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🚄 Infrastructure

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Mar 16, 2026 7:58am

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_ci_optimize (8f56ddb) with main (cd37ee1)}

[greptile-apps]

Greptile Summary

This PR tackles CI pipeline speed (from ~60+ mins → ~11 mins) through three orthogonal efforts: (1) a CircleCI infra overhaul using uv, job parallelism, and timing-based test splitting; (2) xdist-compatible test isolation across multiple conftest.py files (save/restore instead of reload-per-test); and (3) a set of batch-processing bug fixes that needed accompanying test updates. The changes are well-structured and cohesive.

Key changes:

• CI speed: uv pip install replaces serial pip install for requirements.txt, test jobs gain resource_class: medium/large and parallelism: 2, timeout reduced 120m → 15m
• xdist isolation: All major conftest.py files now capture litellm's true defaults at import time and use function-scoped save/restore fixtures rather than importlib.reload per test; module-scoped reload only runs outside xdist workers
• Batch fix (common_utils.py): update_batch_in_database now eagerly sets batch_processed=True on completion and has a graceful fallback for older DB schemas; new resolve_output_file_ids_to_unified resolves raw provider output/error file IDs
• Batch fix (endpoints.py): "complete" (DB-normalised form) added to terminal-state early-return check; resolve_output_file_ids_to_unified called for terminal responses
• Batch fix (check_batch_cost.py): Removes "complete"/"completed" from the status NOT IN exclusion list; the batch_processed=False filter is now the sole guard against re-processing
• Test rewrites: test_eager_tiktoken_load.py moves module-clear tests to subprocesses; router tests switch from shared module-level Router instances to per-test factory functions; mock depth in affinity-check tests moved to async_response_api_handler

Confidence Score: 4/5

• Safe to merge with minor cleanup; the batch fixes are well-tested and the CI changes are opt-in infrastructure improvements
• The CI infrastructure changes are low-risk (additive), the xdist isolation pattern is solid and matches the existing test_litellm/conftest.py blueprint, and the batch fixes are covered by unit tests. One point deducted for the dead code left in test_claude_agent_sdk.py after the refactor (no functional impact but adds confusion) and the slightly misleading safety-net comment in update_batch_in_database. The 15m no_output_timeout is aggressive but the author has empirical evidence (~11m runs) to support it.
• tests/proxy_e2e_anthropic_messages_tests/test_claude_agent_sdk.py (dead code after pytest.fail), litellm/proxy/openai_files_endpoints/common_utils.py (misleading CheckBatchCost safety-net comment)

Important Files Changed

Filename	Overview
.circleci/config.yml	Major CI pipeline overhaul: uses uv for faster installs, adds parallelism/resource_class tweaks, reduces no_output_timeout from 120m→15m, drops several explicit version pins (httpx, anyio, opentelemetry-*) from proxy jobs relying on requirements.txt coverage instead
litellm/proxy/openai_files_endpoints/common_utils.py	Adds resolve_output_file_ids_to_unified helper and eagerly sets batch_processed=True in update_batch_in_database on completion, with a fallback retry path for missing schema columns
litellm/proxy/batches_endpoints/endpoints.py	Adds "complete" to the terminal batch status check and calls the new resolve_output_file_ids_to_unified to resolve raw provider output/error file IDs on retrieve_batch
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Removes "complete"/"completed" from status exclusion list, relying solely on batch_processed=False filter to prevent reprocessing; allows CheckBatchCost to catch batches that transitioned before it ran
tests/proxy_e2e_anthropic_messages_tests/test_claude_agent_sdk.py	Extracts _run_streaming_test helper and adds a 3-retry loop, but the original test body (test_query, received_chunks, full_response, try/except) was left as dead code after pytest.fail()
tests/test_litellm/test_eager_tiktoken_load.py	Replaces in-process module reload (which contaminated xdist workers) with subprocess execution via _run_python; removes the now-unnecessary autouse cleanup fixture
tests/test_litellm/router_utils/pre_call_checks/test_encrypted_content_affinity_check.py	Replaces low-level AsyncHTTPHandler.post mock with BaseLLMHTTPHandler.async_response_api_handler mock and uses typed ResponsesAPIResponse objects; adds num_retries=0 to routers for deterministic test behaviour
tests/local_testing/conftest.py	Adds function-scoped isolate_litellm_state fixture that saves/restores litellm globals before/after each test; demotes litellm reload to module-scope and skips it under xdist
tests/logging_callback_tests/conftest.py	Same xdist-compatible isolation pattern as local_testing/conftest.py; captures true defaults at import time and resets before/after each test; clears _in_memory_loggers to prevent logger instance leaks
tests/local_testing/test_router.py	Rewrites test_aaarouter_dynamic_cooldown_message_retry_time to use direct deployment_callback_on_failure calls instead of patching the OpenAI client; removes real network dependency

Sequence Diagram

sequenceDiagram
    participant C as Client
    participant RB as retrieve_batch endpoint
    participant DB as Database (ManagedObjectTable)
    participant CB as Async Cost Callback (@client)
    participant CBC as CheckBatchCost (background)

    C->>RB: GET /batch/{id}
    RB->>DB: get_batch_from_database(batch_id)
    DB-->>RB: batch (status="complete" or still in-progress)
    alt batch already terminal in DB
        RB->>RB: resolve_input_file_id_to_unified()
        RB->>RB: resolve_output_file_ids_to_unified() [NEW]
        RB-->>C: return cached response
    else batch not yet terminal
        RB->>RB: call provider API (aretrieve_batch)
        RB->>DB: update_batch_in_database(status="complete")
        Note over DB: Sets batch_processed=True [NEW]<br/>Fallback: retry without flag if column missing
        DB-->>RB: updated
        RB->>CB: asyncio.create_task(cost_callback) [fire-and-forget]
        RB-->>C: return response
    end

    CB-->>DB: record batch cost (async, best-effort)

    Note over CBC: Polls periodically
    CBC->>DB: find_many(status NOT IN [failed,expired,cancelled,stale_expired],<br/>batch_processed=False) [CHANGED: complete no longer excluded]
    DB-->>CBC: unprocessed batches
    CBC->>CBC: calculate & record cost
    CBC->>DB: set batch_processed=True

Loading

Comments Outside Diff (1)

1. .circleci/config.yml, line 204-205 (link)

   Aggressive no_output_timeout reduction may cause flaky CI

   The no_output_timeout for both proxy-test jobs is dropped from 120 minutes to 15 minutes. The PR author reports ~11 minute runs on the linked pipeline, giving a 4-minute buffer. That buffer may be too thin if:

   • Dependency installation is a cache miss (uv still needs to download packages)
   • A slow/flaky test causes a single xdist worker to stall

   Consider bumping to 30m as a more comfortable safety margin while still being far below the old 120-minute setting. The same consideration applies to the caching_unit_tests and other jobs that were also reduced.

_{Last reviewed commit: 8f56ddb}

[greptile-apps]

Dead code left after refactor

After pytest.fail(...) raises a Failed exception (it never returns), the entire original test body — test_query, received_chunks, full_response, the try/except block, and the assertions on full_response.strip() — is unreachable dead code. The logic was correctly moved into _run_streaming_test, but the old body was never removed.

Additionally, options (a ClaudeAgentOptions object) is referenced inside the dead block but was only defined inside _run_streaming_test, so if this code were somehow reached it would raise a NameError.

The lines from # Test query onwards should be deleted:

    pytest.fail(
        f"Test failed for {model_name} ({model_description}) after {MAX_RETRIES} attempts: {last_error}"
    )
    # ↑ Everything below this line is dead code and should be removed.

[greptile-apps]

CheckBatchCost safety-net comment is misleading

The inline comment says:

CheckBatchCost acts as a safety net for the rare case where the callback fails.

However, batch_processed=True is set here in update_batch_in_database before the async cost callback has a chance to run. Because CheckBatchCost now queries WHERE batch_processed = FALSE, any batch for which update_batch_in_database succeeds (i.e., the column exists and the write works) will have batch_processed=True and will not be picked up by CheckBatchCost, regardless of whether the cost callback later fails.

The true safety net only applies when update_batch_in_database itself fails to persist batch_processed (e.g., the column is absent and the fallback retry also errors). Consider updating the comment to accurately reflect this narrower guarantee, so future maintainers aren't misled into thinking cost-callback failures are automatically retried:

        # Setting batch_processed=True here unblocks file deletion (which
        # queries batch_processed=False).  NOTE: this flag is set eagerly,
        # so CheckBatchCost is only a safety net for the case where this
        # update itself fails (e.g. old schema without the column).  If the
        # async cost callback fails after this write succeeds, the cost will
        # NOT be re-calculated automatically.
        if db_status == "complete":
            update_data["batch_processed"] = True