build: fix gpu ci

[gwarmstrong]

do not merge until gpu tests pass. trying some things...

Summary by CodeRabbit

• Chores
  • More aggressive Docker cleanup: shortened prune window and added builder-cache pruning to free space sooner.
  • Added automated removal of aged Hugging Face cache (datasets and hub) to reduce storage bloat.
  • HF cache cleanup now runs earlier in the workflow to keep test environments lean.

[coderabbitai]

📝 Walkthrough

Walkthrough

Updated the GPU tests GitHub Actions workflow: renamed a cleanup step, shortened docker system prune window from 360h to 168h, added docker builder prune with 168h, and inserted a pre-checkout step to remove Hugging Face cache (datasets and hub) older than 7 days.

Changes

Cohort / File(s)	Summary
GitHub Actions GPU workflow .github/workflows/gpu_tests.yml	Renamed "Cleanup old containers" → "Cleanup old Docker images and build cache"; changed docker system prune until from 360h to 168h; added docker builder prune with until=168h; added pre-checkout step that deletes HF cache under /mnt/datadrive/nemo-skills-test-data/hf-cache/datasets (depth 2-1) and /mnt/datadrive/nemo-skills-test-data/hf-cache/hub older than 7 days.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The title 'build: fix gpu ci' is vague and generic, lacking specificity about what GPU CI issue is being fixed or how.	Consider a more descriptive title that specifies the actual fix, such as 'build: add HF cache cleanup in GPU CI' or 'build: improve GPU CI cleanup with root permissions'.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch georgea/fix-gpu-ci-03112026

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/gpu_tests.yml:
- Around line 29-32: The BusyBox find in the docker run using image "alpine"
does not support GNU-style "-exec ... {} +" so update the workflow step: either
switch the container image from "alpine" to one with GNU find (e.g., an image
that includes findutils) or rewrite the two find commands to use
BusyBox-compatible "-exec ... {} \;" syntax; also stop swallowing errors by
removing "2>/dev/null" and the trailing "true" so failures surface in CI. Ensure
you update the specific docker run line that invokes find and reference the
exact tokens "alpine", "find", "-exec ... {} +", "2>/dev/null", and "true" when
making the change.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5c1cbc4a-fb0e-4329-b315-9360bc831203

📥 Commits

Reviewing files that changed from the base of the PR and between 600b2ba and 5358a39.

📒 Files selected for processing (1)

• .github/workflows/gpu_tests.yml

[coderabbitai]

⚠️ Potential issue | 🟠 Major

BusyBox find likely makes this cleanup a silent no-op.

alpine uses BusyBox by default, and BusyBox’s documented find syntax only lists -exec CMD ARG ;, not the GNU -exec ... {} + form used on Lines 30-31. Because stderr is redirected to /dev/null and the script ends with true, this step can report success while deleting nothing. Please switch to a BusyBox-compatible form here or use an image with GNU find, and avoid masking unexpected failures. (wiki.alpinelinux.org)

Does Alpine's default BusyBox `find` support `-exec CMD ARG {} +`, or only `-exec CMD ARG {} \;`?

🔧 Proposed fix

-        docker run --rm -v /mnt/datadrive:/mnt/datadrive alpine \
-          sh -c 'find /mnt/datadrive/nemo-skills-test-data/hf-cache/datasets -maxdepth 2 -mindepth 2 -type d -mtime +7 -exec rm -rf {} + 2>/dev/null;
-                 find /mnt/datadrive/nemo-skills-test-data/hf-cache/hub -maxdepth 1 -mindepth 1 -type d -mtime +7 -exec rm -rf {} + 2>/dev/null;
-                 true'
+        docker run --rm -v /mnt/datadrive:/mnt/datadrive alpine \
+          sh -euc '
+            if [ -d /mnt/datadrive/nemo-skills-test-data/hf-cache/datasets ]; then
+              find /mnt/datadrive/nemo-skills-test-data/hf-cache/datasets -maxdepth 2 -mindepth 2 -type d -mtime +7 -exec rm -rf {} \;
+            fi
+            if [ -d /mnt/datadrive/nemo-skills-test-data/hf-cache/hub ]; then
+              find /mnt/datadrive/nemo-skills-test-data/hf-cache/hub -maxdepth 1 -mindepth 1 -type d -mtime +7 -exec rm -rf {} \;
+            fi'

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/gpu_tests.yml around lines 29 - 32, The BusyBox find in
the docker run using image "alpine" does not support GNU-style "-exec ... {} +"
so update the workflow step: either switch the container image from "alpine" to
one with GNU find (e.g., an image that includes findutils) or rewrite the two
find commands to use BusyBox-compatible "-exec ... {} \;" syntax; also stop
swallowing errors by removing "2>/dev/null" and the trailing "true" so failures
surface in CI. Ensure you update the specific docker run line that invokes find
and reference the exact tokens "alpine", "find", "-exec ... {} +",
"2>/dev/null", and "true" when making the change.