[Infra] Reduce llm_translation_testing parallelism and tolerate worker restarts - #25897
[Infra] Reduce llm_translation_testing parallelism and tolerate worker restarts#25897yuneng-berri wants to merge 4 commits into
Conversation
Litellm day 0 opus 4.7 support
Litellm hotfix opus 4.7
…orker restarts Workers in llm_translation_testing have been crashing mid-run with "Not properly terminated" (OOM), even after bumping resource_class to xlarge. Reduce xdist workers from 8 to 4 to lower peak memory, and add --max-worker-restart=5 so a crashed worker is replaced instead of failing the whole run.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Greptile SummaryReduces The actual changeset is a single line in Confidence Score: 5/5Safe to merge — the change is a one-line CI tuning with no production code impact. The PR makes a single, well-motivated change to a CI job configuration. Halving parallelism doubles per-worker RAM (1 GB → 2 GB on resource_class:large) and --max-worker-restart=5 prevents a single OOM crash from aborting the entire suite. No application code is modified. The only non-blocking concern is whether the 15m no_output_timeout remains comfortable with fewer workers, which CI itself will confirm. No files require special attention; .circleci/config.yml is the only changed file.
|
| Filename | Overview |
|---|---|
| .circleci/config.yml | Single-line change: drops xdist workers from 8 → 4 and adds --max-worker-restart=5 on the llm_translation_testing job; resource_class remains large (4 vCPU / 8 GB), giving each worker 2 GB instead of 1 GB. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[llm_translation_testing job
resource_class: large
4 vCPU / 8 GB RAM] --> B[pytest-xdist
-n 4 workers]
B --> W1[Worker 0
~2 GB RAM]
B --> W2[Worker 1
~2 GB RAM]
B --> W3[Worker 2
~2 GB RAM]
B --> W4[Worker 3
~2 GB RAM]
W1 -- OOM crash --> R{Restarts used
< 5?}
W2 -- OOM crash --> R
R -- Yes --> B
R -- No --> FAIL[Job fails]
W1 -- Tests pass --> PASS[Job passes]
W2 -- Tests pass --> PASS
W3 -- Tests pass --> PASS
W4 -- Tests pass --> PASS
Reviews (1): Last reviewed commit: "[Infra] CI: reduce llm_translation_testi..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| done | ||
| uv run --no-sync python -m pytest -v tests/llm_translation $IGNORE_ARGS --junitxml=test-results/junit.xml --durations=20 -n 8 --timeout=120 --timeout_method=thread --retries 2 --retry-delay 5 | ||
| uv run --no-sync python -m pytest -v tests/llm_translation $IGNORE_ARGS --junitxml=test-results/junit.xml --durations=20 -n 4 --timeout=120 --timeout_method=thread --retries 2 --retry-delay 5 --max-worker-restart=5 | ||
| no_output_timeout: 15m |
There was a problem hiding this comment.
no_output_timeout may be tight with half the workers
Halving workers from 8 → 4 will increase wall-clock run time for CPU/memory-bound tests. The current no_output_timeout: 15m is the time without any output; as long as pytest prints progress it won't fire. If any test generates no output for a full 15 minutes (e.g. a hung network call that slips past the 120 s --timeout), the job could be killed. Worth monitoring the first few runs to confirm the suite still comfortably finishes within the overall job limit.
4 participants
Relevant issues
Summary
Problem
Workers in the `llm_translation_testing` CircleCI job have been crashing mid-run with `[gw*] node down: Not properly terminated`, failing the job and blocking the pipeline. The crashes are OOM kills: multiple workers go down near the end of the run (97–99% through), and the log shows accumulating `Unclosed client session` / `Unclosed connector` messages before the final crash. Bumping `resource_class` from `large` to `xlarge` (#25887) did not resolve it.
Fix
Two small, independent mitigations on the pytest invocation:
Testing
CI will exercise the new configuration when this PR runs.
Type
🚄 Infrastructure