[CI] Raise AMD Spec Decode Eagle 1 job timeout to 35min - #55136
Conversation
The MI300 mirror consistently takes ~26min against a 25min limit: all six tests pass, but the last one is SIGTERM'd at ~91% of its GSM8K eval. The ROCm 7.2.3 toolchain wastes ~23s per engine start probing an aiter clang flag it does not support (-amdgpu-coerce-illegal-types), which pushed the job over the edge. Co-authored-by: Kimi <noreply@moonshot.cn> Signed-off-by: JaredforReal <w13431838023@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟢 Approval recommended
Single, low-risk CI configuration change that aligns the AMD mirror timeout with observed runtime and existing mirror timeout patterns.
Pull request overview
Adjusts Buildkite CI timeouts for a single AMD mirror job in the Spec Decode test area to reduce flaky cancellations caused by the job slightly exceeding its current limit.
Changes:
- Increase
:amd: (MI300) Spec Decode Eagle 1: DeepSeek + Qwenmirror timeout from 25 to 35 minutes. - Leave the corresponding NVIDIA job timeout unchanged.
File summaries
| File | Description |
|---|---|
| .buildkite/test_areas/spec_decode.yaml | Raises the AMD mirror timeout for the “Spec Decode Eagle 1: DeepSeek + Qwen” job to avoid CI SIGTERM near completion. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
github-actions
Bot
added
the
ready
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 SummarySummary by CodeRabbit
WalkthroughThe AMD mirror of the “Spec Decode Eagle 1: DeepSeek + Qwen” Buildkite step now has a 35-minute timeout instead of 25 minutes. ChangesSpec Decode Timeout
Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: ⚪ Minimal · up to The AMD Spec Decode CI job now has sufficient time to complete rather than timing out near its expected runtime. No current merge-blocking risk is identified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/ci run |
|
✅ Triggered Buildkite CI #87290 for commit |
…#55136) Signed-off-by: JaredforReal <w13431838023@gmail.com> Co-authored-by: Kimi <noreply@moonshot.cn> Signed-off-by: Jyotirmoy Roy <jyotirmoyroy649@gmail.com>
Purpose
The
:amd: (MI300) Spec Decode Eagle 1: DeepSeek + Qwenjob (25min limit) consistently takes ~25.5-26min and gets SIGTERM'd mid-way through its 6th test (test_eagle_correctness_medium[ROCM_AITER_FA-qwen3_eagle3-transformers], killed at ~91% of its GSM8K eval; the first 5 tests pass with 96-100/100). Observed on two consecutive attempts of build 87038 (25.8/25.4 min).One contributor: ROCm 7.2.3's clang rejects aiter's
-amdgpu-coerce-illegal-types=1probe flag, wasting ~23s per engine start (12 engine starts in this job ≈ 4-5min of pure overhead).Raise the AMD mirror timeout to 35min, in line with the other AMD mirrors (40/50, 45/55, 50/75 vs. their NVIDIA counterparts).
Test Plan
CI config only. Evidence: build 87038 job logs, two attempts, identical timeout signature (
Processed prompts: 91%→Received cancellation signal).Duplicate-work check
No open PR found for this job timeout at the time of writing.
AI assistance
Prepared with AI assistance (Kimi Code); the human submitter has reviewed every changed line.