fix(ci): reserve GPUs for SPUR hardware tests - #137
Conversation
Request AIC_TEST_GRES (gpu:1 by default) for smoke and tiny-test Slurm jobs on every cluster, including SPUR, so hardware CI is scheduled with GPU ownership. Propagate Slurm’s selected visible device into Docker and Compose instead of hardcoding GPU 0, and derive the vLLM container name from that device. Detect a stopped vLLM container during readiness polling and print its logs immediately instead of waiting for the full endpoint timeout.
|
/run-ci-fast |
b018ff0 to
2c72e08
Compare
|
/run-ci-fast |
|
✅ Fast hardware CI passed for ca2b75c (dist-build-fast + smoke-test-fast + tiny-test). This is a single-arch dev check — run |
|
@sbates130272 please take a look -- this fixes a long-standing problem where tiny-test jobs were launched without gpu constraints leading to failures and possibly interfering with others' workloads that they collided with. Also fails faster and cleans up orphaned jobs. I believe that once this merges the other PR #136 will merge cleanly, too. |
|
@amd-ivaganev SPUR was recently updated and only just got suppport for |
sbates130272
left a comment
sbates130272
left a comment
There was a problem hiding this comment.
@amd-ivaganev there are two differnt things here. One is the workflow fix for tiny-test but the rest is a refactor of the .github/scripts/* which are installed on the github runner and which are a seperate topic. Can you fix this PR to only address the tiny-test workflow and then create another PR for the refactor and optomization of the runner scripts?
ca2b75c to
7691223
Compare
|
/run-ci-fast |
Ah, right... The reason for the refactor was this: Re-triggering keeps the same names and since it's done by me and not John the cleanup fails and propagates as tiny-test failure, even though the test itself succeeded... |
|
✅ Fast hardware CI passed for 7691223 (dist-build-fast + smoke-test-fast + tiny-test). This is a single-arch dev check — run |
Do you need me to update the scripts on my runner? Does #138 cover any $USER changes that you need to be made? |
|
@amd-ivaganev I am working on a follow-up PR that addresses similar issues to this PR. We need to pass the devices to Docker in a special way, and a couple of other items. If this PR is still open by the time I am done, I will make my PR against your branch. |
Feel free to drop this PR and replace with your larger changes if it makes it easier. It was a band-aid for the troubles with clearing PR #136 where the superficial errors were blocking the merge of clean code. |
|
@amd-ivaganev sure, if it would be easiest, I can open a PR that includes your changes here as well as mine. But could be good to get any band-aids in to unblock CI. |
Do I read that as "let's merge this now"? |
|
@amd-ivaganev Yes, I think it would be good to merge this now. |
3 participants
Summary
AIC_TEST_GRES(default:gpu:1)Root cause
The SPUR path omitted
--gres=gpu:1, so Slurm could co-locate hardware CI with an existing GPU workload. In the failing run, vLLM found only 16.22 GiB free on a 287.98 GiB device, below the 86.4 GiB requested by--gpu-memory-utilization 0.30. Engine initialization failed, but tiny-test continued polling the unavailable endpoint for the full ten-minute readiness timeout.Scope
This PR intentionally changes only
.slurm/run-build-distribute.sh. The self-hosted runner script refactor, run-attempt worktree isolation, and orphan cancellation handling are split into #138.Validation
bash -n .slurm/run-build-distribute.shgit diff --check--gres=gpu:1ROCR_VISIBLE_DEVICES=0and exposes onegfx950device