Skip to content

fix(ci): extend SGLang GPU test timeout - #12799

Closed
xianlubird wants to merge 1 commit into
ai-dynamo:mainfrom
xianlubird:bug/sglang-gpu-test-timeout
Closed

fix(ci): extend SGLang GPU test timeout#12799
xianlubird wants to merge 1 commit into
ai-dynamo:mainfrom
xianlubird:bug/sglang-gpu-test-timeout

Conversation

@xianlubird

@xianlubird xianlubird commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Increase the SGLang single-GPU test stage timeout from the reusable workflow default of 30 minutes to 45 minutes.
  • Keep the VRAM-aware scheduler and individual pytest timeouts unchanged.
  • Preserve the outer timeout as protection against a genuinely stuck test stage.

Background

Full CI for PR #12746 exposed how little headroom the current SGLang timeout has. In the failed run, the parallel GPU step started at 00:56:52 UTC and was terminated at 01:26:52 UTC, exactly at its 30-minute limit. The runner reported Executing the custom container implementation failed; this was not a pytest assertion failure.

The last scheduled test, tests/router/test_router_e2e_with_sglang.py::test_sglang_kv_router_basic[tcp], did not start until 01:25:22 UTC. Its own timeout was 270 seconds, but the outer step limit stopped the container after roughly 90 seconds. Individual test timeouts already begin when each subprocess starts, so queued time does not consume the test's own budget; the problem is that the workflow-level timeout includes collection, queueing, execution, and cleanup.

This is also close to occurring on successful runs. A recent successful SGLang parallel step took 29 minutes and 33 seconds, leaving only 27 seconds of headroom, while another completed in 13 minutes and 49 seconds. That variation makes a 30-minute stage limit brittle even when every test is healthy.

Implementation

Set gpu_test_timeout_minutes: 45 explicitly for the SGLang test job. This matches the existing vLLM single-GPU timeout and gives a queued test enough time to use its own timeout budget. The value remains an upper bound: successful jobs finish immediately and do not wait for the full 45 minutes.

This does not retry the GPU suite, ignore failures, or change any per-test timeout.

Validation

  • Parsed .github/workflows/pr.yaml with PyYAML.
  • Ran git diff --check.
  • Compared the failed SGLang parallel step with two recent successful runs to verify the timeout boundary and runtime variance.

Summary by CodeRabbit

  • Tests
    • Extended the GPU test job timeout to 45 minutes, allowing longer-running test suites to complete.

Signed-off-by: xianlubird <xianlubird@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@xianlubird
xianlubird temporarily deployed to external_collaborator August 7, 2026 02:12 — with GitHub Actions Inactive
@xianlubird
xianlubird temporarily deployed to external_collaborator August 7, 2026 02:12 — with GitHub Actions Inactive
@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor trusted-contributor Org-External user who is trusted to run CI without Org-member approval labels Aug 7, 2026
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 0945b8d

@xianlubird
xianlubird marked this pull request as ready for review August 7, 2026 02:13
@xianlubird
xianlubird requested a review from a team as a code owner August 7, 2026 02:13

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Comment thread .github/workflows/pr.yaml
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e0d75225-9bc1-46a1-b20c-b7b04b5e16fe

📥 Commits

Reviewing files that changed from the base of the PR and between 6a4b074 and 0945b8d.

📒 Files selected for processing (1)
  • .github/workflows/pr.yaml

Walkthrough

The pull request adds a 45-minute timeout to the sglang-test GPU test job in the pull request workflow.

Changes

GPU test timeout

Layer / File(s) Summary
Configure GPU test timeout
.github/workflows/pr.yaml
The sglang-test job now has a 45-minute GPU test timeout.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers motivation, implementation, and validation, but omits the required Related Issues section and reviewer-start guidance. Add the required Related Issues section with an issue link or checked no-related-issue confirmation, and identify where reviewers should start.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the change to the SGLang GPU test timeout.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@datadog-official

datadog-official Bot commented Aug 7, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 45.14% (-6.19%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0945b8d | Docs | Datadog PR Page | Give us feedback!

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This chang moved qualifying tests from gpu_1 to gpu_0 decreasing test runtime significantly: #12718
If PRs are still experiencing long SGL GPU parallel test runs - they may just have to rebase on / merge main
Example recent run with short SGL test runtime https://github.com/ai-dynamo/dynamo/actions/runs/31057166523/job/92478147343
I recommend closing this PR so we don't bloat our SGL tests runtime by allowing longer running test runs

@xianlubird
xianlubird deleted the bug/sglang-gpu-test-timeout branch August 10, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions external-contribution Pull request is from an external contributor fix size/XS trusted-contributor Org-External user who is trusted to run CI without Org-member approval

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants