Document nightly/inflight official build for CI-failure routing - #36088
Conversation
Adds a deterministic breadcrumb so a bare request like "nightly is broken please fix" routes to the right pipeline. Previously nothing in the repo mapped "nightly"/"inflight feed"/"dogfood feed" to the official signed build, so an agent had to guess its way from a grep to ci-official.yml with no way to know the AzDO definition id or org. - azdo-build-investigator/SKILL.md: add nightly/inflight/dogfood trigger phrases to the description; add a "Nightly / Official Signed Build" section naming the pipeline (dotnet-maui, def 1095, dnceng/internal, ci-official.yml, cron 05:00 UTC on inflight/current), the AzDO investigation recipe, and the recurring vs-workload.props (MSB4019) failure in the Pack Windows "Build Workloads, Sign & Publish" step; add a matching row to the failure-pattern table. - release-readiness/SKILL.md: cross-link the ❌ staleness banner to the investigator section so the banner's "check the nightly pipeline" has a documented target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36088Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36088" |
Skill Validation Results
✅ Skill Validation Results —
|
github-actions
Bot
added
the
area-infrastructure
The child investigation (PR #36089) binlog-proved the Jun 2026 inflight/current nightly outage: PR #32203's blanket netstandard IsPackable=false in Directory.Build.targets clobbered the workload packs' explicit IsPackable=true, so the no--build 'Pack, Sign' step never ran Build → _GenerateVSWorkloadProps never generated vs-workload.props → MSB4019. Record the specific cause, the fix, the inflight-only scope, and the forward-port warning so the breadcrumb teaches the real failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
kubaflo
left a comment
kubaflo
left a comment
There was a problem hiding this comment.
Note
🤖 This review was generated by AI — an automated multi-model code review (Claude Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro).
🔍 AI Multi-Model Code Review — PR #36088
Docs PR (+33/−1, 2 files, base main) — adds nightly/official-build CI-failure routing to two skills.
✅ Verdict: LGTM
Confidence: High — unanimous (Opus high · GPT medium · Gemini high)
Summary
Documents the nightly official signed build (dotnet-maui, def 1095, dnceng/internal) in azdo-build-investigator/SKILL.md — the vs-workload.props MSB4019 failure surface in the Pack Windows job, the root-cause of the Jun 2026 inflight/current outage (PR #32203's blanket IsPackable rule, fixed by #36089), and a new failure-pattern table row — plus a cross-link note in release-readiness/SKILL.md for when the nightly-feed banner turns ❌.
What's good:
- Internally consistent and well-cross-referenced; relative links resolve; the failure-pattern table row is well-formed.
- The root-cause narrative (blanket
IsPackable=falseoverriding the workload packs' explicittrue→_GenerateVSWorkloadPropsAfterTargets=Buildnever running → MSB4019) matches the companion fix in #36089. - Correctly distinguishes the nightly
dotnet-maui(1095) pipeline frommaui-prand from the release pipelines. - Skill validation + static validation pass;
maui-pris path-excluded for.github/**.
💡 Note (non-blocking)
The internal pipeline identity (dotnet-maui / definition 1095 / dnceng-internal) can't be verified from the public repo — it's documentation of internal infrastructure. Worth a maintainer sanity-check that the definition ID and org/project are current, since a stale ID would misroute future investigations. Not a blocker.
🏁 Recommendation
LGTM — accurate, well-structured documentation that pairs with the #36089 fix. Ready for human approval.
Reviewed with the code-review skill: independence-first, full-file reading, repo cross-checking, and CI verification by all 3 models.
kubaflo
left a comment
kubaflo
left a comment
There was a problem hiding this comment.
Note
🤖 AI-generated approval (multi-model review automation, approving on @kubaflo's behalf).
✅ Approving — 3-model LGTM.
Accurate, well-structured documentation of the nightly official signed build (dotnet-maui, def 1095) for CI-failure routing — the vs-workload.props MSB4019 surface, the Jun 2026 inflight/current root-cause (PR #32203's blanket IsPackable rule, fixed by #36089), and a new failure-pattern table row. Internally consistent, links resolve, skill + static validation pass. Pairs with the #36089 fix.
Non-blocking: the internal pipeline identity (def 1095 / dnceng-internal) isn't verifiable from the public repo — worth a maintainer sanity-check that the ID stays current.
Resolves the conflict in azdo-build-investigator/SKILL.md caused by main's refactor to a thin canonical-doc pointer (v3.0). Keeps main's refactored body and appends only this branch's net-new '## Nightly / Official Signed Build (inflight dogfood feed)' routing section, plus merges the nightly trigger phrases into the frontmatter description. The stale inline sections main relocated to .github/docs/maui-ci-facts.md are intentionally dropped. release-readiness/SKILL.md keeps its +2 cross-reference to that new section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR updates MAUI’s internal Copilot skills documentation so that “nightly/inflight/dogfood feed” CI-failure reports route directly to the official signed build pipeline and its known failure modes, rather than the PR validation pipelines.
Changes:
- Expanded
azdo-build-investigatorskill routing phrases to include nightly/inflight/dogfood/offical-build triggers. - Documented the official signed build identity (pipeline
dotnet-maui, definition 1095,dnceng/internal) plus an investigation recipe and a known recurring failure signature (vs-workload.props/MSB4019). - Cross-linked
release-readinessstaleness banner guidance to the investigator skill section.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/release-readiness/SKILL.md | Adds a cross-link from the nightly-feed staleness banner to the documented official-build investigation steps. |
| .github/skills/azdo-build-investigator/SKILL.md | Adds nightly/official-build trigger phrases and a dedicated section documenting the official signed build pipeline and common failure surface. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 3
| - `azdo_search_timeline` (filter `failed`) on that build → identifies the failing job/step + its `logId` | ||
| - `azdo_search_log` on that `logId` → the actual error | ||
|
|
||
| **Known failure surface:** the most common nightly break is the **`Pack Windows`** job → **"Build Workloads, Sign & Publish"** step ([`eng/pipelines/arcade/stage-pack.yml`](../../../eng/pipelines/arcade/stage-pack.yml), building `src/Workload/workloads.csproj`). Recurring signature: |
| ``` | ||
| `vs-workload.props` is generated by the `_GenerateVSWorkloadProps` target in `src/Workload/Microsoft.NET.Sdk.Maui.Manifest/Microsoft.NET.Sdk.Maui.Manifest.csproj` (`AfterTargets="Build"`) during the **preceding** `Pack, Sign` step, then imported by `workloads.csproj`. If it's missing at import time, the manifest pack didn't stage it to `ArtifactsShippingPackagesDir`. Note the **macOS `Pack` job can succeed while `Pack Windows` fails** — a green macOS leg does NOT mean the dogfood feed is healthy. | ||
|
|
||
| **Root cause of the Jun 2026 `inflight/current` outage (binlog-proven, fixed by [#36089](https://github.com/dotnet/maui/pull/36089)):** the workload packs (`Microsoft.NET.Sdk.Maui.Manifest`, `Microsoft.Maui.Sdk`) target **netstandard** but are shipping packages, so they set `<IsPackable>true</IsPackable>` explicitly. PR #32203 ("Don't pack .NET Standard") added a *blanket* `<IsPackable Condition="$(TargetFramework.Contains('netstandard'))">false</IsPackable>` to `Directory.Build.targets` — which is auto-imported at the **end** of every project, so it overrode the packs' explicit `true` → `IsPackable=false`. With `IsPackable=false`, NuGet `Pack` no-ops; and because the **"Pack, Sign" step runs `-pack` *without* `-build`** (only the later workloads step uses `-build`), the `Build` target never ran → `_GenerateVSWorkloadProps` (`AfterTargets="Build"`) never ran → `vs-workload.props` was never written → the line-30 import threw `MSB4019`. The fix guards the blanket rule with `and '$(IsPackable)' == ''` so explicit opt-ins survive. **This regression was `inflight/current`-only** (#32203's commit `543b1ebeb7` is not yet on `main`/`net10.0`) — so when it forward-ports, it must carry the guard or the break reappears. The general lesson when triaging this failure: confirm `IsPackable` actually evaluates `true` for the workload packs in the `pack.binlog`, and that the pack step's `Build` (hence `_GenerateVSWorkloadProps`) actually ran. |
|
|
||
| Determinism / idempotency: the engine captures **one** `UtcNow` per run (`$Data['nightlyFeedNow']`) and reuses it for both the rendered banner and the semantic-hash tier, so a quiet SR tracker still refreshes when the feed crosses a tier boundary, but a same-tier day-count tick does **not** churn the issue. The freshness band is folded into `Get-ReportSemanticHash` (tier|version only — the raw timestamp is never hashed). | ||
|
|
||
| **When the banner is ❌ (feed STALE):** the dogfood bits have stopped flowing because the nightly **official signed build** is failing — pipeline `dotnet-maui` (definition **1095**, org `dnceng` / project `internal`), defined by [`eng/pipelines/ci-official.yml`](../../../eng/pipelines/ci-official.yml) and scheduled daily on `inflight/current`. See the [`azdo-build-investigator`](../azdo-build-investigator/SKILL.md) skill's **Nightly / Official Signed Build (inflight dogfood feed)** section for the investigation recipe and the recurring `vs-workload.props` (`MSB4019`) failure in the `Pack Windows` → "Build Workloads, Sign & Publish" step. |
kubaflo
left a comment
kubaflo
left a comment
There was a problem hiding this comment.
Note
🔍 AI-generated review (automated multi-model orchestrator, on behalf of @kubaflo) — Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro.
✅ Round 2 — APPROVED (merge-refresh re-approval)
The head advanced from 4e258a1f → 7b1badbb, but that new commit is a merge-from-main (Merge branch 'main' into pureween/nightly-pipeline-breadcrumb). The PR's own net diff is byte-identical to the R1-approved content — still just:
.github/skills/azdo-build-investigator/SKILL.md(+30) — new Nightly / Official Signed Build (inflight dogfood feed) section.github/skills/release-readiness/SKILL.md(+2) — cross-link to that section
Content re-verified. The breadcrumb is accurate against known CI facts: nightly pipeline dotnet-maui (def 1095, dnceng/internal), the recurring Pack Windows → vs-workload.props MSB4019 signature, and the #36089 root-cause (blanket IsPackable=false from #32203 overriding the workload packs' explicit opt-in on inflight/current). No inaccuracies.
CI: maui-pr = skipping (by-design .github/** path-exclusion) — for a pure-docs PR the review is the gate. license/cla pass.
Docs-only, no behavioral risk. Re-approving on the refreshed head.
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description
Docs/skill-only change. Adds a deterministic breadcrumb so a bare request like "nightly is broken please fix" routes an agent (or a human) straight to the right pipeline and the known failure surface, instead of guessing.
Today nothing in the repo maps "nightly" / "inflight feed" / "dogfood feed" to the official signed build.
copilot-instructions.mddocuments only the PR pipelines (maui-pr,-devicetests,-uitests), andazdo-build-investigatormentionedci-official.ymlonly tangentially (CFSClean/Gradle). So an agent told the dogfood feed is stale had no way to know the AzDO definition id (1095) or org (dnceng/internal) to even query the builds.This matters now because we just shipped the nightly dogfood feed staleness banner (#36066) — when that banner turns ❌, the natural next question is "which pipeline do I go fix?" and the repo couldn't answer it.
What changed
.github/skills/azdo-build-investigator/SKILL.mdnightly/inflight feed/dogfood feed/official buildtrigger phrases to the skilldescription(fixes routing).dotnet-maui, definition 1095,dnceng/internal, defined byeng/pipelines/ci-official.yml, cron05:00 UTConinflight/current), the AzDO investigation recipe (azdo_buildsbydefinitionId+branch →azdo_search_timeline→azdo_search_log), and the recurringvs-workload.props(MSB4019) failure in thePack Windows→ "Build Workloads, Sign & Publish" step. Plus a note that the macOS leg can pass while Windows fails, and that the release pipelines are distinct..github/skills/release-readiness/SKILL.mdGrounding
Every fact was confirmed against the live failing run — definition
1095/dnceng-internal, theci-official.ymlschedule,stage-pack.yml:150("Build Workloads, Sign & Publish"), and theMSB4019: vs-workload.props was not foundsignature oninflight/current(a multi-day streak as of this writing). No product/runtime code is touched.Risk
Documentation only. No code paths, no tests affected.
ci-official.ymlexcludes.github/**, andmaui-prskips.github/**-only PRs by design.Update — confirmed root cause folded in
A companion investigation (fix PR #36089, base
inflight/current) binlog-proved the actual cause of the current outage, now documented in the breadcrumb:Microsoft.NET.Sdk.Maui.Manifest,Microsoft.Maui.Sdk) target netstandard but ship, so they set<IsPackable>true</IsPackable>explicitly.<IsPackable Condition="$(TargetFramework.Contains('netstandard'))">false</IsPackable>inDirectory.Build.targets(auto-imported last) overrode that →IsPackable=false.-packwithout-build, soBuildnever ran →_GenerateVSWorkloadProps(AfterTargets="Build") never wrotevs-workload.props→ the line-30 import threwMSB4019.inflight/current-only (the Don't pack .NET Standard #32203 commit isn't onmain/net10.0yet) — forward-port must carry theand '$(IsPackable)' == ''guard or the break reappears.The breadcrumb now teaches the real failure mechanism, not just the symptom, and links the fix.