[Infra] Reduce llm_translation_testing parallelism and tolerate worker restarts

[yuneng-berri]

Relevant issues

Summary

Problem

Workers in the `llm_translation_testing` CircleCI job have been crashing mid-run with `[gw*] node down: Not properly terminated`, failing the job and blocking the pipeline. The crashes are OOM kills: multiple workers go down near the end of the run (97–99% through), and the log shows accumulating `Unclosed client session` / `Unclosed connector` messages before the final crash. Bumping `resource_class` from `large` to `xlarge` did not resolve it.

Fix

Two small, independent mitigations on the pytest invocation:

• Drop xdist workers from `-n 8` to `-n 4`. Memory pressure per worker is the dominant failure mode, not CPU throughput; these tests are I/O-bound.
• Add `--max-worker-restart=5` so a crashed worker is replaced and its remaining tests are redistributed, instead of failing the entire job on the first OOM.

Staging counterpart of #25897.

Testing

CI will exercise the new configuration when this PR runs.

Type

🚄 Infrastructure

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 16, 2026 8:12pm

[greptile-apps]

Greptile Summary

This PR reduces the llm_translation_testing CircleCI job's xdist parallelism from -n 8 to -n 4 and adds --max-worker-restart=5 to tolerate OOM-induced worker crashes, following a prior resource_class: large → xlarge bump that did not fully resolve the issue. Both mitigations are targeted and consistent with the I/O-bound nature of these tests.

Confidence Score: 5/5

Safe to merge — minimal, targeted CI config change with no impact on application code or test logic.

Single CI config file changed; only the pytest invocation for llm_translation_testing is modified (worker count halved, restart tolerance added). Both changes directly address the described OOM failure mode. CI supply-chain rules (pinned uv version, checksum verified) are already satisfied and untouched. No P0/P1 findings.

No files require special attention.

Important Files Changed

Filename	Overview
.circleci/config.yml	Reduces llm_translation_testing xdist workers from -n 8 to -n 4 and adds --max-worker-restart=5 to tolerate OOM-induced worker crashes

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[llm_translation_testing job starts\nresource_class: xlarge] --> B[Install deps\nuv 0.10.9 + checksum verified]
    B --> C[pytest -n 4\n--timeout=120\n--max-worker-restart=5\n--retries 2]
    C --> D{Worker crash?}
    D -- No --> E[Test completes normally]
    D -- Yes, restarts ≤ 5 --> F[Pending tests redistributed\nto remaining workers]
    F --> D
    D -- Yes, restarts > 5 --> G[Job fails]
    E --> H[Store JUnit results]

Loading

_{Reviews (1): Last reviewed commit: "[Infra] CI: reduce llm_translation_testi..." | Re-trigger Greptile}