Skip to content

[CI] Retry CPU image builds after truncated network streams - #56938

Closed
khluu wants to merge 1 commit into
vllm-project:mainfrom
khluu:agent/fix-cpu-build-network-retry
Closed

khluu wants to merge 1 commit into
vllm-project:mainfrom
khluu:agent/fix-cpu-build-network-retry

Conversation

@khluu

@khluu khluu commented Sep 15, 2026

Copy link
Copy Markdown
Member

Summary

  • classify truncated Git/HTTP stream failures as transient in the CPU image build retry loop
  • cover GitHub clone disconnect/early-EOF/index-pack failures and interrupted LLVM downloads
  • preserve the existing four-attempt limit and 10/20/40-second backoff

Why

Main builds #88996 and #88997 lost six CPU jobs on one physical host before tests. Five image builds ended while cloning triton-cpu with HTTP/2 stream cancellation followed by fetch-pack disconnect, early EOF, and invalid index-pack output. The sixth lost its LLVM archive stream with BrokenPipeError: [Errno 32] Broken pipe.

run-cpu-test.sh already retries transient image-build network errors, but none of those concrete transport signatures matched BUILD_RETRY_PATTERN, so every affected job exited after attempt 1 instead of using the existing bounded retry policy. The exact PR build for each merged change had passed the relevant lanes, and these failures are unrelated to either source diff.

Duplicate check

Searched open PRs and issues for CPU build retry HTTP/2 early EOF, triton-cpu unexpected disconnect retry, and CPU CI triton clone early EOF; no existing fix was found.

Tests

  • bash -n .buildkite/scripts/hardware_ci/run-cpu-test.sh
  • matched all five observed transport signatures against the amended pattern and verified a representative compiler error does not match
  • uvx --from shellcheck-py shellcheck .buildkite/scripts/hardware_ci/run-cpu-test.sh
  • PRE_COMMIT_HOME=/tmp/sherlock-precommit-cpu-retry uvx --python 3.12 pre-commit run --files .buildkite/scripts/hardware_ci/run-cpu-test.sh
  • git diff --check

No model evaluation is applicable; this changes CI retry classification only.

This draft was prepared with AI assistance. A human submitter must review every changed line and validate the retry behavior before marking it ready.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
@khluu

khluu commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #89003 for commit 7b962d3f2c21.

@mergify mergify Bot added ci/build cpu Related to CPU backends labels Sep 15, 2026
@khluu

khluu commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

Closing this duplicate draft: the existing draft #56597 already owns CPU image-download retry classification. I will fold the newly observed HTTP/2 clone-disconnect signatures into #56597 instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant