Skip to content

CI: Do not start PR pipelines for draft pull requests - #11932

Merged
Alexey-Rivkin merged 1 commit into
openucx:masterfrom
Alexey-Rivkin:ci-no-draft-builds
Oct 13, 2026
Merged

Alexey-Rivkin merged 1 commit into
openucx:masterfrom
Alexey-Rivkin:ci-no-draft-builds

Conversation

@Alexey-Rivkin

@Alexey-Rivkin Alexey-Rivkin commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

What?

Do not start CI for draft PRs.

Why?

Azure behaviour change - now marking a draft PR "ready for review" makes Azure queue a second run of the same commit. This causes extra full runs, adding load to already strained resources.
Microsoft has no fix planned (https://developercommunity.visualstudio.com/t/11032682).

How?

buildlib/azure-pipelines-pr.yml:
drafts: false in the pr: block. One line.

@Alexey-Rivkin
Alexey-Rivkin marked this pull request as ready for review September 9, 2026 12:41
@Alexey-Rivkin Alexey-Rivkin changed the title CI: Do not start the PR pipeline for draft pull requests CI: Do not start PR pipelines for draft pull requests Sep 9, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread buildlib/azure-pipelines-release.yml Resolved
Comment thread buildlib/azure-pipelines-perf.yml Resolved
Comment thread buildlib/azure-pipelines-pr.yml Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage gap: per REVIEW.md, CI changes should demonstrate that the stated behavior actually changed. PR triggers are evaluated from the source branch YAML, so this is verifiable by toggling this PR itself to draft and confirming no pipeline queues, then back to ready and confirming the runs start. Worth asking for that evidence if it isn't already in the PR discussion.

roiedanino
roiedanino previously approved these changes Sep 9, 2026
@Alexey-Rivkin

Copy link
Copy Markdown
Contributor Author

/azp run UCX PR

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread buildlib/azure-pipelines-pr.yml Resolved
Marking a draft PR "ready for review" makes Azure queue a second run of
the same commit and auto-cancel the run already in progress. Over the
last 30 days this happened 26 times, 17 of them killing a running draft
build (one after 18 hours in the test queue). Azure has no option to
build drafts and skip that transition, so stop building drafts instead:
a draft gets its first PR pipeline run when it is marked ready.

Only the PR pipeline is gated. The snapshot and release pipelines do
real work on a PR only when asked via an "AZP/SNAPSHOT: " or
"AZP/RELEASE: " commit title, which is how packaging changes get tested
from draft PRs; their unrequested runs cost seconds, so they stay as is.

Checked on this PR: pushes while draft queued no UCX PR run, an
"/azp run UCX PR" comment on the draft was refused by Azure, and marking
the PR ready queued the run.

Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread buildlib/azure-pipelines-pr.yml Resolved
Comment thread buildlib/azure-pipelines-pr.yml Resolved
@Alexey-Rivkin
Alexey-Rivkin marked this pull request as ready for review September 9, 2026 13:13
@svc-ucx

svc-ucx commented Sep 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 706a1646

TL;DR: The AddressSanitizer "roce on worker 0" job was killed mid-compile by an Azure DevOps agent shutdown ("The agent has received a shutdown signal" → "The operation was canceled"), not by any UCX code/test failure — simply re-run the pipeline.

Full analysis

Summary: Job AddressSanitizer roce on worker 0 (build 135293) aborted while ./contrib/test_jenkins.sh was still compiling gtest sources; no test ever ran.

Root cause: Infrastructure-level cancellation, not a code defect. The log shows continuous, healthy build progress — object files completing every 1–15 s from 14:20:04 through 14:28:28 (largest gap ≈ 30 s, between 14:22:26 and 14:22:45), so nothing hung and nothing hit a wall-clock timeout. At 14:28:23 the agent emitted ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled. followed by ##[error]The Operation will be canceled and ##[error]The operation was canceled, then Finishing: Run ./contrib/test_jenkins.sh and Finalize Job. That message is produced by the self-hosted agent host going away (VM/container recycled, pool drained, or the run superseded/canceled), not by the build step failing. The last compile lines (uct/v2/test_uct_query.cc, uct/gtest-test_uct_perf.o) are unrelated bystanders. Consistent with this, the head commit on the branch, 706a1646 "CI: Do not start the PR pipeline for draft pull requests" (Alexey Rivkin), only adds drafts: false to the pr: trigger in buildlib/azure-pipelines-pr.yml:6 — it cannot affect a compilation step, though it does mean that toggling the PR to draft / pushing a new commit will cancel in-flight runs like this one.

Implicated commit: none — build was not failed by code. Branch head is [REDACTED:Hex High Entropy String] (Alexey Rivkin), which only edits the PR trigger.

File: buildlib/azure-pipelines-pr.yml:4-6 (only the PR-trigger change under test; no source file implicated in the abort)

Suggested fix: Re-queue the PR pipeline (or push an empty commit) — this is a transient agent loss. If it recurs on the same agent, ask CI ops to check the health/drain schedule of the self-hosted pool hosting the roce on worker 0 job (spot/scale-set recycling and container restarts produce exactly this message). Since the PR under test changes PR-trigger semantics, also confirm the run wasn't cancelled by the PR being switched to draft or by a superseding push before treating it as an agent problem; no UCX code change is warranted.

Related: PR #11932 (this PR, ci-no-draft-builds); prior trigger change 1ebb4fa "AZP: Exclude .ci/ from PR pipeline trigger (#11439)". No matching issue for the agent-shutdown signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f7b8c526-a469-45b9-89a6-cc7135af1198 in the triage console for the audit trail.

@gleon99 gleon99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼

@Alexey-Rivkin
Alexey-Rivkin merged commit fc262d7 into openucx:master Oct 13, 2026
162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants