ci: pin uv to 0.11.19 (0.11.20 resolver regression breaks the test suite) by wprazuch · Pull Request #1576 · NVIDIA-NeMo/Gym

wprazuch · 2026-06-11T10:04:46Z

Problem

The Full test suite (Test) job has been red on every PR, failing 7 servers
(math_with_code, newton_bench, arena_judge, reasoning_gym, ether0, aviary,
stirrup_agent) with ModuleNotFoundError at test collection — scipy, scikit-learn,
matplotlib, PIL.

These deps are declared (and pinned) in each server's requirements.txt. The real cause:

uv 0.11.20 (released 2026-06-10) has a resolver regression — it silently drops pinned
direct dependencies from uv pip install -r requirements.txt when the requirements also
include an editable -e install (as every server's -e nemo-gym[dev] @ ../../ does). No error,
no conflict — the package is just omitted.

CI installs uv unpinned (curl -LsSf https://astral.sh/uv/install.sh | sh), so it picked up
0.11.20 the day it released — which is exactly when the suite started failing. The first
(passing) run used an earlier uv.

Evidence (reproduced locally with the exact CI install command)

For resources_servers/reasoning_gym (source .venv/bin/activate && uv pip install -r requirements.txt openai==2.7.2):

uv version	Resolved	`matplotlib==3.10.6`
0.11.19	154 packages	✅ installed → `import matplotlib` OK
0.11.20	150 packages	❌ dropped → `ModuleNotFoundError`

Bisected 0.10.2 → 0.11.20: every version through 0.11.19 works; only 0.11.20 is broken.

Fix

Pin the uv installer to 0.11.19 (latest known-good) in full-test-suite.yml (Test + wheel
jobs) and unit-tests.yml, with a comment explaining why. Once uv ships a fix, the pin can be
bumped.

Not caused by — and unblocks — any PR that triggers the full matrix (e.g. #1561, #1575). Worth
also reporting the regression upstream to astral-sh/uv.

uv 0.11.20 (released 2026-06-10) has a resolver regression: it silently drops pinned direct dependencies from `uv pip install -r requirements.txt` when the requirements also include an editable `-e` install. This dropped matplotlib/scipy/scikit-learn/Pillow from 7 servers' per-server test venvs, reding the full-test-suite Test job on every PR. CI installed uv unpinned (curl .../uv/install.sh), so it picked up 0.11.20 the day it released. Pin to 0.11.19, the latest known-good version. Verified locally with the exact CI install command for resources_servers/reasoning_gym: - uv 0.11.19: resolves 154 packages incl. matplotlib==3.10.6 -> import OK - uv 0.11.20: resolves 150 packages, matplotlib dropped -> ModuleNotFoundError Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

copy-pr-bot · 2026-06-11T10:04:49Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

ko3n1g · 2026-06-12T11:17:57Z

/ok to test 8ed2d8e

ko3n1g · 2026-06-12T11:21:15Z

/ok to test cc06fa7

…n → ~2-3min) (#1577) ## Problem The **Unit tests → `Test`** check runs `ng_test_all`, which iterates all ~114 server modules **strictly sequentially** in a single runner (build venv → pytest → delete venv per module), taking **~22 min**. This is the long pole on every PR that touches core files (which trigger the full suite via the existing change-detection). ## Change **`ng_test_all` sharding** (`nemo_gym/cli.py`): new `+num_shards` / `+shard_index` options. Modules are partitioned **round-robin over a sorted list** so heavy modules (longmt_eval, the scientific servers) spread evenly across shards. The `fail_on_total_and_test_mismatch` check still runs against the full (unsharded) module list. Extracted `_select_shard()` with unit tests. **Workflow restructure** (`.github/workflows/unit-tests.yml`): - `detect` — change classification (unchanged logic), exposed as job outputs. - `Test` — core library unit tests (`ng_dev_test`) + the changed-servers path. (Fast now — the full server loop moved out.) - `Server suite (shard N)` — `ng_test_all +num_shards=8 +shard_index=N` across an **8-way matrix** (`fail-fast: false`), only on a full run. - `Server suite` — aggregation job that gates on all shards (single required check). **Result: full-suite wall-time ~22 min → ~2-3 min.** Same total compute, parallelized. Also pins uv to **0.11.19** (0.11.20 has a resolver regression that drops pinned deps — see #1576; this overlaps #1576 on `unit-tests.yml`, so merge #1576 first or fold them). ## ⚠️ Branch-protection action required Server tests moved out of the `Test` job into the sharded jobs. To preserve gating, add **`Server suite`** as a required status check alongside **`Test`**. (`Server suite` uses `if: always()` so it reports green when the full suite isn't needed for a change, and red if any shard fails — so it's safe to require unconditionally.) ## Testing - `_select_shard` unit tests: complete + disjoint partition, balanced sizes, round-robin stride, out-of-range guard. `pytest tests/unit_tests/test_cli.py` 14/14, ruff clean, workflow YAML parses. - The workflow itself can only be fully validated once it runs on CI. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

marta-sd · 2026-06-12T12:10:58Z

/ok to test 3be810d

ko3n1g · 2026-06-12T12:12:05Z

/ok to test 3be810d

Unblocks CI on this martas/1434-stacked branch: pin uv (0.11.20 drops pinned deps -> scipy/sklearn/matplotlib/PIL ModuleNotFoundError on 7 servers; = #1576) and pull main's graphwalks data/example_rollouts.jsonl (fixes its data validation). Drop both when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Shared CI fixes for the martas/1434-stacked CLI work: pin uv (0.11.20 drops pinned deps -> 7 servers fail; = #1576) and pull main's graphwalks example_rollouts.jsonl (fixes its data validation). This branch is the base for the ng_validate (#1599) and config-error (#1609) PRs so the fixes live in one place. Drop when martas/1434 rebases on main. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…n → ~2-3min) (#1577) The **Unit tests → `Test`** check runs `ng_test_all`, which iterates all ~114 server modules **strictly sequentially** in a single runner (build venv → pytest → delete venv per module), taking **~22 min**. This is the long pole on every PR that touches core files (which trigger the full suite via the existing change-detection). **`ng_test_all` sharding** (`nemo_gym/cli.py`): new `+num_shards` / `+shard_index` options. Modules are partitioned **round-robin over a sorted list** so heavy modules (longmt_eval, the scientific servers) spread evenly across shards. The `fail_on_total_and_test_mismatch` check still runs against the full (unsharded) module list. Extracted `_select_shard()` with unit tests. **Workflow restructure** (`.github/workflows/unit-tests.yml`): - `detect` — change classification (unchanged logic), exposed as job outputs. - `Test` — core library unit tests (`ng_dev_test`) + the changed-servers path. (Fast now — the full server loop moved out.) - `Server suite (shard N)` — `ng_test_all +num_shards=8 +shard_index=N` across an **8-way matrix** (`fail-fast: false`), only on a full run. - `Server suite` — aggregation job that gates on all shards (single required check). **Result: full-suite wall-time ~22 min → ~2-3 min.** Same total compute, parallelized. Also pins uv to **0.11.19** (0.11.20 has a resolver regression that drops pinned deps — see #1576; this overlaps #1576 on `unit-tests.yml`, so merge #1576 first or fold them). Server tests moved out of the `Test` job into the sharded jobs. To preserve gating, add **`Server suite`** as a required status check alongside **`Test`**. (`Server suite` uses `if: always()` so it reports green when the full suite isn't needed for a change, and red if any shard fails — so it's safe to require unconditionally.) - `_select_shard` unit tests: complete + disjoint partition, balanced sizes, round-robin stride, out-of-range guard. `pytest tests/unit_tests/test_cli.py` 14/14, ruff clean, workflow YAML parses. - The workflow itself can only be fully validated once it runs on CI. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ite) (#1576) ## Problem The **Full test suite** (`Test`) job has been red on every PR, failing 7 servers (`math_with_code`, `newton_bench`, `arena_judge`, `reasoning_gym`, `ether0`, `aviary`, `stirrup_agent`) with `ModuleNotFoundError` at test collection — `scipy`, `scikit-learn`, `matplotlib`, `PIL`. These deps **are** declared (and pinned) in each server's `requirements.txt`. The real cause: **`uv 0.11.20` (released 2026-06-10) has a resolver regression** — it silently drops pinned direct dependencies from `uv pip install -r requirements.txt` when the requirements also include an editable `-e` install (as every server's `-e nemo-gym[dev] @ ../../` does). No error, no conflict — the package is just omitted. CI installs uv **unpinned** (`curl -LsSf https://astral.sh/uv/install.sh | sh`), so it picked up 0.11.20 the day it released — which is exactly when the suite started failing. The first (passing) run used an earlier uv. ## Evidence (reproduced locally with the exact CI install command) For `resources_servers/reasoning_gym` (`source .venv/bin/activate && uv pip install -r requirements.txt openai==2.7.2`): | uv version | Resolved | `matplotlib==3.10.6` | |---|---|---| | 0.11.19 | 154 packages | ✅ installed → `import matplotlib` OK | | **0.11.20** | 150 packages | ❌ dropped → `ModuleNotFoundError` | Bisected 0.10.2 → 0.11.20: every version **through 0.11.19 works**; only **0.11.20** is broken. ## Fix Pin the uv installer to **0.11.19** (latest known-good) in `full-test-suite.yml` (Test + wheel jobs) and `unit-tests.yml`, with a comment explaining why. Once uv ships a fix, the pin can be bumped. Not caused by — and unblocks — any PR that triggers the full matrix (e.g. #1561, #1575). Worth also reporting the regression upstream to astral-sh/uv. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch requested a review from a team as a code owner June 11, 2026 10:04

wprazuch mentioned this pull request Jun 11, 2026

ci: parallelize the server test suite (in-process concurrency, ~17min → faster locally + CI) #1577

Merged

ko3n1g previously approved these changes Jun 12, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public June 12, 2026 11:18 Inactive

ko3n1g enabled auto-merge (squash) June 12, 2026 11:18

copy-pr-bot Bot temporarily deployed to public June 12, 2026 11:19 Inactive

copy-pr-bot Bot temporarily deployed to public June 12, 2026 11:20 Inactive

Merge branch 'main' into wprazuch/pin-uv-ci

cc06fa7

copy-pr-bot Bot temporarily deployed to public June 12, 2026 11:21 Inactive

copy-pr-bot Bot temporarily deployed to public June 12, 2026 11:23 Inactive

Merge branch 'main' into wprazuch/pin-uv-ci

3be810d

marta-sd dismissed ko3n1g’s stale review via 3be810d June 12, 2026 12:10

copy-pr-bot Bot temporarily deployed to public June 12, 2026 12:11 Inactive

ko3n1g approved these changes Jun 12, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public June 12, 2026 12:12 Inactive

copy-pr-bot Bot temporarily deployed to public June 12, 2026 12:13 Inactive

ko3n1g merged commit 3397b75 into main Jun 12, 2026
28 of 29 checks passed

marta-sd deleted the wprazuch/pin-uv-ci branch June 30, 2026 11:28

Navigation Menu

Uh oh!

ci: pin uv to 0.11.19 (0.11.20 resolver regression breaks the test suite) - #1576

ci: pin uv to 0.11.19 (0.11.20 resolver regression breaks the test suite)#1576
ko3n1g merged 3 commits into
mainfrom
wprazuch/pin-uv-ci

wprazuch commented Jun 11, 2026

Uh oh!

copy-pr-bot Bot commented Jun 11, 2026

Uh oh!

ko3n1g commented Jun 12, 2026

Uh oh!

ko3n1g commented Jun 12, 2026

Uh oh!

marta-sd commented Jun 12, 2026

Uh oh!

ko3n1g commented Jun 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wprazuch commented Jun 11, 2026

Problem

Evidence (reproduced locally with the exact CI install command)

Fix

Uh oh!

copy-pr-bot Bot commented Jun 11, 2026

Uh oh!

ko3n1g commented Jun 12, 2026

Uh oh!

ko3n1g commented Jun 12, 2026

Uh oh!

marta-sd commented Jun 12, 2026

Uh oh!

ko3n1g commented Jun 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants