Repository navigation
[CI/Build] Retry CPU image stream interruptions - #56597
Merged
Merged
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
tlrmchlsmth
approved these changes
Sep 12, 2026
khluu
marked this pull request as draft
September 14, 2026 04:27
khluu
changed the title
Recognize the Python downloader signature emitted when the Triton CPU LLVM artifact stalls so the existing bounded Docker-build retry loop can recover instead of failing immediately. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Recognize the downloader BrokenPipeError emitted by the Triton CPU LLVM artifact fetch so the existing bounded Docker-build retry loop can recover. Co-authored-by: Sherlock <sherlock@openai.com> Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
khluu
force-pushed
the
fix/cpu-build-read-timeout-retry
branch
from
September 15, 2026 03:37
4268071 to
3248f77
Compare
khluu
changed the title
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #89005 for commit |
Member
Author
|
Current-head validation: Buildkite #89005 resolved literal head This head covers the eight pre-test failures now observed on physical |
khluu
marked this pull request as ready for review
September 15, 2026 04:18
ItsRoy69
pushed a commit
to ItsRoy69/vllm
that referenced
this pull request
Sep 15, 2026
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Sherlock <sherlock@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
keneoneth
pushed a commit
to keneoneth/vllm
that referenced
this pull request
Sep 16, 2026
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Sherlock <sherlock@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TimeoutError: The read operation timed outBrokenPipeError: [Errno 32] Broken pipetriton-cpuclone:fetch-packunexpected disconnectWhy
This extends the retry logic introduced by #55317. Main #88385 first exposed Python read timeouts in two CPU shards; both exact retries passed unchanged. Main #88718 CPU Language shard 1 later lost the same 1.8-GiB LLVM download with
BrokenPipeError, but the classifier rejected it and exited after attempt 1.The same gap widened on Sep 15. Main #88996 lost three CPU jobs on
aicf-jf-cicd-01: twotriton-cpuclones ended in HTTP/2 cancellation/early EOF, while one LLVM download ended inBrokenPipeError. Main #88997 then lost all three CPU Language shards on the same host at the same instant with the identical HTTP/2 clone-disconnect chain. Main #88999 added an eighth pre-test failure on the same host when another LLVM download broke its pipe. Each job reached the existing classifier, none matched, and each exited after attempt 1.These eight Sep 15 failures occurred before tests. The exact PR build for each merged source change had already passed the relevant CPU lanes, so the shared transport signature is not attributable to any of the three source diffs.
Why this doesn't duplicate an open PR
Searched open vLLM PRs and issues for CPU image-build read timeouts, broken pipes, truncated HTTP/2 streams, Triton CPU LLVM downloads, and
BUILD_RETRY_PATTERN. No other open change covers these signatures. A briefly opened duplicate draft #56938 was closed before validation, and its distinct stream signatures were folded into this existing owner.Tests
bash -n .buildkite/scripts/hardware_ci/run-cpu-test.shBUILD_RETRY_PATTERNuvx --from shellcheck-py shellcheck .buildkite/scripts/hardware_ci/run-cpu-test.shPRE_COMMIT_HOME=/tmp/sherlock-precommit-cpu-retry uvx --python 3.12 pre-commit run --files .buildkite/scripts/hardware_ci/run-cpu-test.shgit diff --checkNo model evaluation is applicable; this changes CI retry classification only. Exact CPU PR validation of current head
3248f7752fis pending; the external failure paths cannot be forced deterministically.AI assistance disclosure
OpenAI Codex assisted with the diagnosis, pattern extensions, and validation. The earlier timeout change was reviewed by the human submitter; a human maintainer must review every newly amended line before this draft is marked ready.