fix(ci): extend SGLang GPU test timeout - #12799
Conversation
Signed-off-by: xianlubird <xianlubird@gmail.com>
|
/ok to test 0945b8d |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughThe pull request adds a 45-minute timeout to the ChangesGPU test timeout
Estimated code review effort: 1 (Trivial) | ~2 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
|
🎯 Code Coverage (details) 🔗 Commit SHA: 0945b8d | Docs | Datadog PR Page | Give us feedback! |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
This chang moved qualifying tests from gpu_1 to gpu_0 decreasing test runtime significantly: #12718
If PRs are still experiencing long SGL GPU parallel test runs - they may just have to rebase on / merge main
Example recent run with short SGL test runtime https://github.com/ai-dynamo/dynamo/actions/runs/31057166523/job/92478147343
I recommend closing this PR so we don't bloat our SGL tests runtime by allowing longer running test runs
Summary
Background
Full CI for PR #12746 exposed how little headroom the current SGLang timeout has. In the failed run, the parallel GPU step started at 00:56:52 UTC and was terminated at 01:26:52 UTC, exactly at its 30-minute limit. The runner reported
Executing the custom container implementation failed; this was not a pytest assertion failure.The last scheduled test,
tests/router/test_router_e2e_with_sglang.py::test_sglang_kv_router_basic[tcp], did not start until 01:25:22 UTC. Its own timeout was 270 seconds, but the outer step limit stopped the container after roughly 90 seconds. Individual test timeouts already begin when each subprocess starts, so queued time does not consume the test's own budget; the problem is that the workflow-level timeout includes collection, queueing, execution, and cleanup.This is also close to occurring on successful runs. A recent successful SGLang parallel step took 29 minutes and 33 seconds, leaving only 27 seconds of headroom, while another completed in 13 minutes and 49 seconds. That variation makes a 30-minute stage limit brittle even when every test is healthy.
Implementation
Set
gpu_test_timeout_minutes: 45explicitly for the SGLang test job. This matches the existing vLLM single-GPU timeout and gives a queued test enough time to use its own timeout budget. The value remains an upper bound: successful jobs finish immediately and do not wait for the full 45 minutes.This does not retry the GPU suite, ignore failures, or change any per-test timeout.
Validation
.github/workflows/pr.yamlwith PyYAML.git diff --check.Summary by CodeRabbit