Skip to content

Add agentic labeler workflow for issues and PRs#35382

Merged
PureWeen merged 9 commits into
mainfrom
feature/agentic-labeler
May 15, 2026
Merged

Add agentic labeler workflow for issues and PRs#35382
PureWeen merged 9 commits into
mainfrom
feature/agentic-labeler

Conversation

@JanKrivanek
Copy link
Copy Markdown
Member

@JanKrivanek JanKrivanek commented May 11, 2026

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description

Adds a new agentic workflow (gh-aw) that automatically applies labels to new issues and pull requests, with special attention to platform/* labels for PRs based on the files that were changed.

Inspired by githubnext/agentics issue-triage, but scoped down to labeling only — it does not post analysis comments, does not close issues, and does not communicate directly with users.

Triggers

  • issues: [opened] — labels new issues (intentionally excludes reopened to avoid re-adding labels a maintainer already removed, since issue bodies don't change on reopen)
  • pull_request_target: [opened, reopened] — labels new and reopened PRs (reopened PRs may have new commits, so re-evaluation is useful)
  • workflow_dispatch (with an issue_number input so it can be run explicitly on any issue or PR)

Labeling rules

  • Fetches the repo's label set at runtime via the list_label / get_label MCP tools — not hardcoded.
  • Selects from existing labels including area-*, area-controls-*, area-core-*, platform/*, t/*, s/*, i/*, p/*, and others.
  • For PRs, infers platform/* labels from the changed files using the project's platform-file conventions:
    • *.android.cs, /Platform/Android/, /Platforms/Android/platform/android
    • *.ios.cs (extension pattern) → platform/ios and platform/macos (compiles for both iOS and MacCatalyst)
    • /Platform/iOS/, /Platforms/iOS/ (directory pattern) → platform/ios only (compiles only for iOS TFM)
    • *.maccatalyst.cs, /Platform/MacCatalyst/platform/macos only
    • *.windows.cs, /Platform/Windows/platform/windows
    • *.tizen.cs, /Tizen/platform/tizen
  • Conservative by default: if nothing clearly applies, the agent calls noop instead. One add_labels call allowed per run (max: 1).

Security model

  • Read-only agent — permissions are contents: read, issues: read, pull-requests: read. The agent runs inside a sandboxed container with no write credentials.
  • Safe-output writes — label application happens in a separate safe-output job with write permissions, capped at 1 call.
  • roles: all — allows community contributors' issues/PRs to be labeled. Safe because the agent is read-only and the only write surface is add_labels.
  • min-integrity: none — allows the MCP gateway to return content from all authors (including first-time contributors), so the agent can read the body it needs to label.
  • Prompt-injection guardrails — explicit instructions telling the agent to ignore labeling instructions in issue/PR bodies, never use an item_number from untrusted text, and derive labels only from technical content and file paths.
  • Noise suppressionnoop, missing-tool, report-incomplete, and report-failure are all configured to not create tracker issues.

Files

  • .github/workflows/agentic-labeler.md — the agentic workflow source
  • .github/workflows/agentic-labeler.lock.yml — compiled GitHub Actions YAML (generated by gh aw compile, v0.68.3)

Notes for reviewers

  • This is consistent with the other gh-aw workflows in the repo (ci-doctor, copilot-evaluate-tests, daily-repo-status).
  • Draft because we may want to validate behavior on a few real issues/PRs (via workflow_dispatch) before enabling on every new issue/PR.

Adds .github/workflows/agentic-labeler.md (gh-aw) that runs on new issues, new pull requests, and on manual workflow_dispatch. It chooses appropriate area-* and platform/* labels from the existing repo label set. For PRs, platform/* labels are inferred from the changed file paths (.android.cs, /Platform/iOS/, .maccatalyst.cs, etc.) per the repo's platform-file conventions.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 11, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 35382

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 35382"

@JanKrivanek JanKrivanek marked this pull request as ready for review May 11, 2026 17:13
Copilot AI review requested due to automatic review settings May 11, 2026 17:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new gh-aw agentic workflow intended to automatically apply existing repo labels to newly opened/reopened issues and pull requests, with PR platform labels inferred from the set of changed files.

Changes:

  • Introduces an agentic workflow prompt (agentic-labeler.md) defining labeling scope, allowed labels, and platform inference rules.
  • Adds the compiled GitHub Actions workflow (agentic-labeler.lock.yml) generated via gh aw compile, including safe-output plumbing for label writes.
Show a summary per file
File Description
.github/workflows/agentic-labeler.md Defines the labeling agent prompt, allowed labels list, and platform label inference rules for PRs.
.github/workflows/agentic-labeler.lock.yml Compiled workflow implementing the agent run + safe-output label application and related control-plane behaviors.

Copilot's findings

Comments suppressed due to low confidence (4)

.github/workflows/agentic-labeler.md:139

  • The prompt instructs the model to use the add-labels safe-output, but in compiled workflows the actual safe-output tool name is add_labels (underscore) (see .github/workflows/agentic-labeler.lock.yml:224 and the safe outputs config around add_labels). If the model follows this text literally, it may emit an invalid tool call and no labels will be applied. Recommend updating the wording/examples in the Output section to explicitly call add_labels and show the expected JSON shape (item_number, labels, optional repo).
    .github/workflows/agentic-labeler.lock.yml:903
  • The conclusion job is set to treat noop as an issue-worthy event (GH_AW_NOOP_REPORT_AS_ISSUE: "true"). If the agent ever emits a noop (or if the platform injects one), this workflow may create a new GitHub issue instead of just doing nothing, which is outside the stated “labeling only” scope. Suggest disabling noop issue reporting (e.g., configure safe-outputs: noop: { report-as-issue: false } in the source .md and recompile).
          GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }}
          GH_AW_NOOP_MAX: "1"
          GH_AW_WORKFLOW_NAME: "Agentic Labeler"
          GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
          GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }}
          GH_AW_NOOP_REPORT_AS_ISSUE: "true"
        with:

.github/workflows/agentic-labeler.lock.yml:946

  • The workflow is configured to open issues when the agent reports missing tools or incomplete execution (GH_AW_MISSING_TOOL_CREATE_ISSUE / GH_AW_REPORT_INCOMPLETE_CREATE_ISSUE are true). That can create tracker noise unrelated to labeling and contradicts the “does not communicate directly with users” intent. Recommend turning off issue creation for these reporting paths (where supported by gh-aw frontmatter) and rely on run artifacts/logs for diagnostics instead.
        env:
          GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }}
          GH_AW_MISSING_TOOL_CREATE_ISSUE: "true"
          GH_AW_WORKFLOW_NAME: "Agentic Labeler"
        with:
          github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}
          script: |
            const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs');
            setupGlobals(core, github, context, exec, io, getOctokit);
            const { main } = require('${{ runner.temp }}/gh-aw/actions/missing_tool.cjs');
            await main();
      - name: Record incomplete
        id: report_incomplete
        uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9
        env:
          GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }}
          GH_AW_REPORT_INCOMPLETE_CREATE_ISSUE: "true"
          GH_AW_WORKFLOW_NAME: "Agentic Labeler"

.github/workflows/agentic-labeler.lock.yml:975

  • Agent failures are configured to be reported as GitHub issues (GH_AW_FAILURE_REPORT_AS_ISSUE: "true"). For a high-frequency trigger like issues/opened and pull_request_target/opened, transient Copilot/MCP outages could generate a burst of issues. Consider disabling failure-as-issue reporting for this workflow and keep failures confined to the workflow run logs/artifacts.
          GH_AW_GROUP_REPORTS: "false"
          GH_AW_FAILURE_REPORT_AS_ISSUE: "true"
          GH_AW_TIMEOUT_MINUTES: "15"
  • Files reviewed: 2/2 changed files
  • Comments generated: 3

Comment thread .github/workflows/agentic-labeler.md Outdated
Comment thread .github/workflows/agentic-labeler.lock.yml Outdated
Comment thread .github/workflows/agentic-labeler.lock.yml Outdated
@JanKrivanek JanKrivanek reopened this May 11, 2026
@github-actions github-actions Bot added the area-tooling XAML & C# Hot Reload, XAML Editor, Live Visual Tree, Live Preview, Debugging label May 11, 2026
…y contributors, use exact safe-output tool name

- safe-outputs: disable noop/missing-tool/report-incomplete/failure issue creation (was contradicting the labeling-only contract)
- on: roles: all (workflow is safe to run for community contributors since the agent is read-only and writes go through the sandboxed safe-output job)
- prompt: refer to the safe-output tool by its exact name add_labels
@kubaflo
Copy link
Copy Markdown
Contributor

kubaflo commented May 11, 2026

🔍 Multi-Model Review — Agentic Labeler Workflow

Reviewed by Claude Opus 4.7, Claude Opus 4.6, and GPT-5.5 in parallel. Below is the synthesized findings, deduplicated and severity-ranked.


🔴 High: gh label list will fail inside the agent sandbox

All three models flagged this independently.

The prompt (step 3) instructs the agent to:

Fetch the current list of labels with gh label list --limit 300 (bash) and choose from that list.

But the frontmatter only declares tools.github.toolsets: [default]no bash tool is enabled. Per the repo's gh-aw instructions, the agent container has all credentials scrubbed, so gh CLI won't be authenticated even if bash were available.

When label discovery fails, the agent will either hallucinate labels from the prompt examples or skip listing entirely.

Suggested fix: Replace the gh label list instruction with a reference to the MCP list_labels tool (e.g., "Use the list_labels tool to fetch the repository's current labels"). This matches the pattern of the other gh-aw workflows in the repo.


🟡 Medium: Overlap with existing triage.yml — dual labeling on new issues

The repo already has .github/workflows/triage.yml that triggers on issues: [opened] and runs AI-powered triage for platform/*, area-*, and regression labels. Currently its label-application is gated behind workflow_dispatch or /triage comment (with a TODO: delete this condition when we are ready!). When that gate is eventually removed, both AI systems will label every new issue independently, potentially with conflicting results.

Even today, find-similar-issues.yml also triggers on issues: [opened], adding a third concurrent workflow per new issue.

Suggested fix: Document the migration plan (e.g., "this replaces triage.yml") or add mutual exclusion.


🟡 Medium: reopened trigger re-adds labels after manual removal

The workflow is additive-only (add_labels). When an issue is closed and reopened:

  1. Agent originally applied platform/android
  2. Maintainer removed platform/android (it was wrong)
  3. Issue reopened → agent re-adds platform/android

There's no mechanism to detect or respect manual label removals.

Suggested fix: Consider removing reopened from triggers, or add prompt instructions to check existing labels and skip ones that were already present (though this is hard to detect reliably).


🟡 Medium: workflow_dispatch may silently skip labeling

(GPT-5.5 finding) — The add_labels safe-output tool may require item_number for workflow_dispatch triggers (since there's no triggering issue/PR context). The prompt doesn't explicitly instruct the agent to include item_number in the add_labels call. If gh-aw can't infer the target, the label operation is silently skipped.

Suggested fix: Explicitly instruct: "When calling add_labels, always pass item_number."


🟡 Medium: Overly broad /Android/ and /Windows/ path patterns

The platform detection rules include bare /Android/ and /Windows/ path segments. These match non-source paths like:

  • src/Templates/src/templates/maui-mobile/.template.config/Windows/
  • Test infrastructure or docs under generic /Android/ paths

The established convention in the codebase uses /Platform/Android/ or /Platforms/Android/ — the bare /Android/ is unnecessarily broad.

Suggested fix: Either tighten patterns to known platform source directories only, or add a note in the prompt clarifying these are heuristics and the agent should use judgment.


🟡 Medium: noop should be required, not optional, when no labels apply

(GPT-5.5 finding) — The prompt says "consider calling noop" when no labels apply. If gh-aw safe-output workflows require a safe-output call to signal completion, an agent that simply stops without calling any safe-output tool may be treated as an incomplete or failed run.

Suggested fix: Change "consider calling noop" to "call noop with a short reason".


🔵 Low: platform/maccatalyst label exists but is never referenced

The repo has both platform/macos (description: "macOS / Mac Catalyst") and platform/maccatalyst (no description). The prompt maps all MacCatalyst-related files to platform/macos only. This seems intentional given the platform/macos description, but the agent's live list_labels call will surface platform/maccatalyst — it may incorrectly apply it.

Suggested fix: Either delete the platform/maccatalyst label if it's deprecated, or add a note in the prompt: "Use platform/macos for MacCatalyst (not platform/maccatalyst)."


✅ Non-issues (verified correct)

  • Label casing: The repo's platform labels ARE lowercase (platform/ios, platform/macos) — no mismatch with the prompt (Opus 4.7 verified via gh label list).
  • pull_request_target + roles: all: Safe — agent has read-only permissions, writes go through sandboxed safe-outputs.
  • cancel-in-progress: false: Correct for additive labeling — two runs on the same issue adding the same labels is harmless.
  • Missing platform/linux patterns: Intentional — MAUI doesn't have .linux.cs platform-file conventions.
  • Compiler version v0.68.3: Matches the other workflows in the repo.
  • Issue creation paths disabled: All four report-as-issue/create-issue knobs correctly set to false (addressed in commit 5).
  • add_labels vs add-labels naming: Fixed in commit 5.

Review performed by Claude Opus 4.7 + Claude Opus 4.6 + GPT-5.5 in parallel, synthesized by Claude Opus 4.6 (1M).

…n platform paths, drop reopened on issues

- Replace 'gh label list' instruction with the list_labels MCP tool (the agent sandbox has no authenticated gh CLI)
- Require explicit item_number on every add_labels call (works under workflow_dispatch where there is no event context)
- Require noop call when no labels apply (signals the run completed intentionally)
- Tighten platform path patterns to /Platform(s)?/<Name>/ only; drop bare /Android/, /iOS/, /Windows/, /MacCatalyst/, /macOS/, /Tizen/ (template/docs paths matched too broadly)
- Remove 'reopened' from issues trigger to avoid re-adding labels a maintainer has manually removed (PRs keep reopened — different lifecycle)
- Replace non-existent label examples (bug, regression, proposal) with real ones (t/bug, t/enhancement, i/regression, s/needs-repro, p/0-3)
@JanKrivanek
Copy link
Copy Markdown
Member Author

@kubaflo — thanks for the multi-model review! Walking through each item with what was verified and what changed (commit 08ac9c7):


🔴 High: gh label list will fail inside the agent sandbox

... no bash tool is enabled ... gh CLI won't be authenticated even if bash were available.
Suggested fix: Replace the gh label list instruction with a reference to the MCP list_labels tool

Fixed. The prompt now uses the list_labels MCP tool from the github toolset, with an explicit "do not try gh label list from a shell" callout.


🟡 Medium: Overlap with existing triage.yml — dual labeling on new issues

... find-similar-issues.yml also triggers on issues: [opened] ...

Not applicable. Verified — neither .github/workflows/triage.yml nor .github/workflows/find-similar-issues.yml exist in the repo (the multi-model reviewers appear to have hallucinated them; possibly seeing them in another repo). No overlap to mitigate. No fix needed.


🟡 Medium: reopened trigger re-adds labels after manual removal

Fixed for issues. Removed reopened from the issues: trigger so the workflow only labels at creation, respecting any subsequent manual label removals. Kept reopened for pull_request_target because PR reopens are a distinct lifecycle (typically the PR was closed by mistake or by a stale-bot) and the changed-file set may have shifted, so re-evaluation is desirable.


🟡 Medium: workflow_dispatch may silently skip labeling

The add_labels safe-output tool may require item_number for workflow_dispatch triggers ...
Suggested fix: Explicitly instruct: "When calling add_labels, always pass item_number."

Fixed. The "Target" section now tells the agent to determine the target item number from the trigger or inputs.issue_number and pass it as item_number on every add_labels call. The "Output" section repeats the requirement.


🟡 Medium: Overly broad /Android/ and /Windows/ path patterns

Fixed. Tightened the platform path patterns to only /Platform/<Name>/ and /Platforms/<Name>/. Dropped bare /Android/, /iOS/, /Windows/, /MacCatalyst/, /macOS/, /Tizen/ (those matched template, docs, and tooling paths). Also explicitly noted in the prompt to not match on bare segments.


🟡 Medium: noop should be required, not optional, when no labels apply

Fixed. Changed "consider calling noop" to "required to signal that the workflow ran to completion intentionally without labeling" with a one-sentence reason.


🔵 Low: platform/maccatalyst label exists but is never referenced

Not applicable. Verified via gh label listplatform/maccatalyst does not exist in the repo. The only relevant labels are platform/macos, platform/ios, platform/android, platform/windows, platform/linux, platform/tizen. The current mapping (MacCatalyst → platform/macos) is the only valid choice. No fix needed.


Bonus fix: Replaced bogus label examples in the prompt

While verifying the labels above, I noticed the prompt's "other useful labels" examples (bug, regression, proposal, enhancement, documentation) don't exist in this repo either. Replaced with real ones: t/bug, t/enhancement, t/docs, t/breaking, i/regression, s/needs-repro, s/needs-info, s/duplicate, p/0p/3. This matters because even though the agent fetches labels live, mismatched examples in the prompt can prime it toward the wrong names.


Summary

Finding Severity Action
gh label list won't work 🔴 High Fixed → use list_labels MCP tool
Overlap with triage.yml 🟡 Medium N/A — files don't exist
reopened re-adds labels 🟡 Medium Fixed for issues, kept for PRs
workflow_dispatch item_number 🟡 Medium Fixed → explicit instruction
Bare /Android/ /Windows/ 🟡 Medium Fixed → tightened patterns
noop required 🟡 Medium Fixed → required
platform/maccatalyst 🔵 Low N/A — label doesn't exist
Bogus label examples (bonus) Fixed → real label names

Workflow recompiles cleanly. Thanks again for the thorough review!

@JanKrivanek
Copy link
Copy Markdown
Member Author

/azp run maui-pr-uitests, maui-pr-devicetests

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

@kubaflo
Copy link
Copy Markdown
Contributor

kubaflo commented May 13, 2026

🔍 Multi-Model Re-Review — Post Commit 6

Re-reviewed by Claude Opus 4.7, Claude Opus 4.6, and GPT-5.5 after commit 08ac9c7 ("Address multi-model review: use list_labels MCP, require noop, tighten platform paths, drop reopened on issues").


✅ Previous findings — verification

# Finding Status
1 🔴 gh label list fails in sandbox ⚠️ Attempted fix, but new issue — see below
2 🟡 triage.yml overlap Non-issuetriage.yml no longer exists in the repo
3 🟡 reopened re-adds labels Fixed for issues (removed); PRs keep reopened — acceptable design choice
4 🟡 workflow_dispatch missing item_number Fixed — explicit instruction + output section
5 🟡 Broad platform patterns Fixed — tightened to /Platform(s)?/<Name>/
6 🟡 noop optional Fixed — now mandatory
7 🔵 platform/maccatalyst confusion Non-issueplatform/macos is canonical per its description

🔴 Critical: list_labels tool is unavailable — wrong name AND wrong toolset

Unanimous across all three models.

The fix for Finding #1 replaced gh label list with "Fetch the current list of labels using the list_labels MCP tool." This fails on two levels:

  1. Wrong tool name. The github-mcp-server's actual tool is list_label (singular), not list_labels.

  2. Tool not in enabled toolset. The frontmatter declares toolsets: [default], which expands to context, repos, issues, pull_requests. The list_label tool lives in the labels toolset, which is not included in default.

At runtime, the agent will get a tool-not-found error and have no way to discover the repository's label set. It will fall back to guessing from the prompt examples — which themselves have incorrect names (see next finding).

Suggested fix:

  • Frontmatter: toolsets: [default, labels]
  • Prompt: change list_labelslist_label

🟡 Medium-High: list_label only returns first 100 of ~440 labels (no pagination)

(Opus 4.6) — Even after fixing the toolset/name, the list_label MCP tool's underlying GraphQL query uses labels(first: 100) with no pagination cursor. dotnet/maui has ~440 labels. The agent would only see the first ~23% (likely alphabetical), missing many area-*, platform/*, and status labels.

Suggested fix: This is an upstream github-mcp-server limitation. Consider adding a note in the prompt that the agent should call list_label multiple times or validate individual label names with get_label if uncertain. Alternatively, enumerate key label prefixes/families in the prompt as a fallback.


🟡 Medium: Four label examples use incorrect names (missing emoji suffixes)

Unanimous across all three models.

Prompt says Actual repo label
t/enhancement t/enhancement ☀️
t/docs t/docs 📝
t/breaking t/breaking 💥
s/duplicate s/duplicate 2️⃣

GitHub's label API requires exact string match including emoji. The other 8 examples (t/bug, i/regression, s/needs-repro, s/needs-info, p/0p/3) are correct.

Suggested fix: Update to exact names with emoji, or drop these four and let the agent discover them via list_label.


🟡 Medium: s/needs-info and s/needs-repro trigger repo automation on PRs

(GPT-5.5) — The prompt allows applying s/needs-info and s/needs-repro to any item. But repo policy automation (resourceManagement.yml) treats these as issue-only on PRs: it replaces s/needs-info with s/pr-needs-author-input and removes s/needs-repro, posting comments in both cases. The agent could trigger unintended bot comments.

Suggested fix: Add a note: "On PRs, use s/pr-needs-author-input instead of s/needs-info. Do not apply s/needs-repro to PRs."


✅ Verified correct / non-issues

  • /AndroidNative/ path: Valid — src/Core/AndroidNative/ exists in the repo
  • Lock file: Recompiled with matching v0.68.3 compiler; hash updated
  • reopened asymmetry (issues: removed, PRs: kept): Reasonable design choice — PRs are more likely to need re-evaluation after reopen
  • Platform pattern tightening: Correctly scoped to /Platform/<Name>/ and /Platforms/<Name>/ conventions

Review by Claude Opus 4.7 + Claude Opus 4.6 + GPT-5.5, synthesized by Claude Opus 4.6 (1M).

…emoji label names, PR-specific status guidance

- toolsets: add 'labels' so the list_label / get_label MCP tools are available (default toolset doesn't include them)
- prompt: rename list_labels -> list_label (singular, per github-mcp-server source)
- prompt: warn about list_label's 100-item cap (no pagination); recommend get_label for verification when picking labels not surfaced in the listing
- prompt: use exact label names with emoji suffixes for t/enhancement, t/docs, t/breaking, s/duplicate
- prompt: add PR-specific caveat — don't apply s/needs-info or s/needs-repro to PRs (repo automation rewrites/removes them); use s/pr-needs-author-input instead
@JanKrivanek
Copy link
Copy Markdown
Member Author

@kubaflo — thanks for the re-review! All four findings verified against github-mcp-server source and the live repo state. Fixes in commit 045ad5a.


🔴 Critical: list_labels tool is unavailable — wrong name AND wrong toolset

  1. Wrong tool name. The github-mcp-server's actual tool is list_label (singular), not list_labels.
  2. Tool not in enabled toolset. default expands to context, repos, issues, pull_requests. The list_label tool lives in the labels toolset.

Suggested fix: Frontmatter toolsets: [default, labels]; prompt list_labelslist_label.

Fixed. Verified in labels.go:

// ListLabels lists labels from a repository
func ListLabels(t translations.TranslationHelperFunc) inventory.ServerTool {
    return NewTool(
        ToolsetLabels,
        mcp.Tool{
            Name:        "list_label",
            ...

Both the toolset (ToolsetLabels) and the singular name (list_label) match your finding exactly. Compiled lock file now contains GITHUB_TOOLSETS: "context,repos,issues,pull_requests,labels".


🟡 Medium-High: list_label only returns first 100 of ~440 labels (no pagination)

Confirmed and worked around. Verified the source — the GraphQL query is hard-coded labels(first: 100) with no cursor. Verified gh label list --limit 1000 returns 440 labels for dotnet/maui, so the agent will only see ~23%.

Fix in prompt: added an explicit pagination caveat instructing the agent to:

  1. Treat the list_label listing as incomplete.
  2. Trust the label families enumerated in the prompt (area-*, platform/*, t/*, s/*, i/*, p/*) as reliable guides since they're documented here.
  3. Use get_label (which IS registered in the issues toolset too — see GetLabelForLabelsToolset) to verify any candidate label name not surfaced in the listing.

I considered upstreaming a pagination fix to github-mcp-server, but that's a separate workstream — the prompt-level workaround is sufficient for now.


🟡 Medium: Four label examples use incorrect names (missing emoji suffixes)

Prompt says Actual repo label
t/enhancement t/enhancement ☀️
t/docs t/docs 📝
t/breaking t/breaking 💥
s/duplicate s/duplicate 2️⃣

Fixed. Verified all four by enumerating the live label list. Updated to exact strings including emoji. Also added an explicit instruction in the prompt: "using exact names including any emoji suffixes."


🟡 Medium: s/needs-info and s/needs-repro trigger repo automation on PRs

... resourceManagement.yml ... replaces s/needs-info with s/pr-needs-author-input and removes s/needs-repro, posting comments in both cases.

Suggested fix: "On PRs, use s/pr-needs-author-input instead of s/needs-info. Do not apply s/needs-repro to PRs."

Fixed. Verified — resourceManagement.yml line 254-255 explicitly "Replace s/needs-info with s/pr-needs-author-input for PRs", and line 549 confirms an automated reply is posted. Added a "PR-specific status caveat" in the prompt with the exact wording you suggested.


Summary

Finding Severity Verified in Action
list_labels wrong name + toolset 🔴 Critical pkg/github/labels.go Fixed → [default, labels] + list_label
100-label pagination cap 🟡 Med-High labels(first: 100) in source; 440 labels in repo Documented + recommend get_label
Missing emoji suffixes (4 labels) 🟡 Medium live label list Fixed → exact names
s/needs-* triggers PR automation 🟡 Medium resourceManagement.yml:254 Fixed → use s/pr-needs-author-input on PRs

Lock file recompiles cleanly. Thanks for catching the toolset gap — that one would have silently broken the entire workflow at runtime.

@kubaflo
Copy link
Copy Markdown
Contributor

kubaflo commented May 13, 2026

🔍 Multi-Model Re-Review — Post Commit 7 (045ad5a)

Re-reviewed by Claude Opus 4.7, Claude Opus 4.6, and GPT-5.5 after commit 045ad5a ("Address re-review: enable labels toolset, fix list_label name, exact emoji label names, PR-specific status guidance").


✅ All four previous findings — properly fixed

# Finding Verification
1 🔴 list_labels wrong name + toolset ✅ Fixed → list_label (singular) + toolsets: [default, labels]. Lock.yml confirmed: GITHUB_TOOLSETS: "context,repos,issues,pull_requests,labels". Verified against github-mcp-server source.
2 🟡 100-label pagination cap ✅ Documented with practical workaround — use get_label for verification of candidates not in the listing. Label family prefixes in the prompt serve as a reliable fallback.
3 🟡 Four emoji-suffixed labels wrong ✅ Fixed — all four now exact: t/enhancement ☀️, t/docs 📝, t/breaking 💥, s/duplicate 2️⃣. Verified against live repo.
4 🟡 s/needs-info/s/needs-repro on PRs ✅ Fixed — PR-specific caveat added: use s/pr-needs-author-input instead.

✅ New label examples — all verified to exist

All 7 new examples (t/native-embedding, t/desktop, t/a11y, s/needs-attention, s/no-repro, s/not-a-bug, s/pr-needs-author-input) confirmed present in the live repo via gh label list.

✅ Tool availability verified

  • list_label — available via labels toolset ✅
  • get_label — available via both labels AND issues (default) toolsets ✅

🟢 No new issues found

All three models independently verified the changes and found no significant issues. The PR is in good shape for merge.


Review by Claude Opus 4.7 + Claude Opus 4.6 + GPT-5.5, synthesized by Claude Opus 4.6 (1M).

@JanKrivanek
Copy link
Copy Markdown
Member Author

/azp run maui-pr-uitests, maui-pr-devicetests

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

Copy link
Copy Markdown
Member

@PureWeen PureWeen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial PR Review + gh-aw Guide Evaluation

Methodology: 3 independent AI reviewers with adversarial consensus (unanimous → include, 2/3 → include at lower severity, 1/3 → follow-up dispute round). Additionally evaluated against the gh-aw guide reference and the Test-GhAwWorkflowSecurity.ps1 automated scanner.

The core architecture is sound — read-only agent with sandboxed safe-output for label writes. The platform-label logic correctly handles the .ios.cs → both iOS + MacCatalyst convention. The main concerns are around blast-radius caps, integrity filtering, and prompt injection hardening.


Findings Summary

# Sev Finding Consensus Source
1 Cross-item item_number injection 2/3 Adversarial
2 ⚠️ roles: all + pull_request_target prompt injection 2/3 + scanner CRITICAL Both
3 ⚠️ max: 10 should be max: 1 gh-aw guide gh-aw guide
4 ⚠️ min-integrity conflict — agent may not read community content gh-aw guide gh-aw guide
5 ⚠️ Issue trigger missing reopened (PR desc says it's there) 3/3 after dispute Adversarial
6 ⚠️ Platform table row ambiguity (/Platforms/iOS/ dir vs .ios.cs) 3/3 after dispute Adversarial
7 ⚠️ reopened trigger may re-add labels a maintainer removed 2/3 after dispute Adversarial
8 💡 missing-tool/report-incomplete config keys silently dropped 2/3 Adversarial
9 💡 Compiler v0.68.3 → recompile with v0.72.1 for security fixes gh-aw guide gh-aw guide

CI Status

maui-pr, maui-pr-devicetests, maui-pr-uitests are all skipping (expected — workflow-only change). license/cla ✅, add-dogfood-comment ✅.


Upgrade Opportunity

Recompiling with gh-aw v0.72.1 (current: v0.68.3) would pick up fixes for && expression corruption, safe-outputs permission regression, conclusion comment accuracy, over-billing, and firewall binary version. Also consider adding checkout: false (saves ~10-30s runner time since this workflow only uses MCP/API tools) and engine: { max-turns: 10 } to cap token usage.

Comment thread .github/workflows/agentic-labeler.md Outdated
Comment thread .github/workflows/agentic-labeler.md
Comment thread .github/workflows/agentic-labeler.md Outdated
Comment thread .github/workflows/agentic-labeler.md
Comment thread .github/workflows/agentic-labeler.md
Comment thread .github/workflows/agentic-labeler.md Outdated
Comment thread .github/workflows/agentic-labeler.md
Comment thread .github/workflows/agentic-labeler.lock.yml Outdated
…ection guardrails, split iOS table

- safe-outputs.add-labels.max: 10 -> 1 (matches 'exactly once' prompt; bounds blast radius if injected)
- tools.github.min-integrity: none (required pairing with roles:all so MCP returns content from FIRST_TIME_CONTRIBUTOR / CONTRIBUTOR users)
- Prompt: add explicit prompt-injection guardrails section; never read item_number from untrusted text; ignore any labeling instructions in body/comments
- Prompt: split the iOS table row — file-extension *.ios.cs compiles for both iOS+MacCatalyst; /Platform(s)/iOS/ directory compiles only for iOS TFM
- Prompt: clarify the 'do not follow body instructions' rule in 'What NOT to do'
- Restored missing-tool/report-incomplete create-issue:false (these ARE honored — verified suppressing them re-introduces GH_AW_*_CREATE_ISSUE: true)

Deliberately not changed:
- issues.types: kept [opened] (no reopened) — re-evaluating reopened issues would re-add labels a maintainer manually removed; documented intent in description
- compiler version v0.68.3 — matches all other gh-aw workflows in this repo; cross-cutting upgrade should be a separate PR
PureWeen pushed a commit that referenced this pull request May 13, 2026
### Context

Add ability for maintainers to trigger the AzDO PR review pipeline via
`/review` comment on PR

### Notes

- The workflow allways runs from main - so users cannot chage behavior
in their PRs
- Unprivileged users slash command is ignored
- The 'agentic-labeler.md‎' pipeline referenced in comments of this
pipeline is being added by #35382

### Tested execution:

- GitHub Actions run:
https://github.com/dotnet/maui/actions/runs/25163585137

- DevDiv pipeline run:
https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=13980704

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@JanKrivanek
Copy link
Copy Markdown
Member Author

/azp run maui-pr-uitests, maui-pr-devicetests

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

@PureWeen
Copy link
Copy Markdown
Member

Suggested PR description update

The description has drifted from the implementation after several review rounds. Here is a suggested replacement for everything below the [!NOTE] block:


Description

Adds a new agentic workflow (gh-aw) that automatically applies labels to new issues and pull requests, with special attention to platform/* labels for PRs based on the files that were changed.

Inspired by githubnext/agentics issue-triage, but scoped down to labeling only — it does not post analysis comments, does not close issues, and does not communicate directly with users.

Triggers

  • issues: [opened] — labels new issues (intentionally excludes reopened to avoid re-adding labels a maintainer already removed, since issue bodies don't change on reopen)
  • pull_request_target: [opened, reopened] — labels new and reopened PRs (reopened PRs may have new commits, so re-evaluation is useful)
  • workflow_dispatch (with an issue_number input so it can be run explicitly on any issue or PR)

Labeling rules

  • Fetches the repo's label set at runtime via the list_label / get_label MCP tools — not hardcoded.
  • Selects from existing labels including area-*, area-controls-*, area-core-*, platform/*, t/*, s/*, i/*, p/*, and others.
  • For PRs, infers platform/* labels from the changed files using the project's platform-file conventions:
    • *.android.cs, /Platform/Android/, /Platforms/Android/platform/android
    • *.ios.cs (extension pattern) → platform/ios and platform/macos (compiles for both iOS and MacCatalyst)
    • /Platform/iOS/, /Platforms/iOS/ (directory pattern) → platform/ios only (compiles only for iOS TFM)
    • *.maccatalyst.cs, /Platform/MacCatalyst/platform/macos only
    • *.windows.cs, /Platform/Windows/platform/windows
    • *.tizen.cs, /Tizen/platform/tizen
  • Conservative by default: if nothing clearly applies, the agent calls noop instead. One add_labels call allowed per run (max: 1).

Security model

  • Read-only agent — permissions are contents: read, issues: read, pull-requests: read. The agent runs inside a sandboxed container with no write credentials.
  • Safe-output writes — label application happens in a separate safe-output job with write permissions, capped at 1 call.
  • roles: all — allows community contributors' issues/PRs to be labeled. Safe because the agent is read-only and the only write surface is add_labels.
  • min-integrity: none — allows the MCP gateway to return content from all authors (including first-time contributors), so the agent can read the body it needs to label.
  • Prompt-injection guardrails — explicit instructions telling the agent to ignore labeling instructions in issue/PR bodies, never use an item_number from untrusted text, and derive labels only from technical content and file paths.
  • Noise suppressionnoop, missing-tool, report-incomplete, and report-failure are all configured to not create tracker issues.

Files

  • .github/workflows/agentic-labeler.md — the agentic workflow source
  • .github/workflows/agentic-labeler.lock.yml — compiled GitHub Actions YAML (generated by gh aw compile, v0.68.3)

Notes for reviewers

  • This is consistent with the other gh-aw workflows in the repo (ci-doctor, copilot-evaluate-tests, daily-repo-status).
  • Draft because we may want to validate behavior on a few real issues/PRs (via workflow_dispatch) before enabling on every new issue/PR.

Move the portable labeling rules (area-matching, platform-file
conventions, label discovery) to .github/skills/agentic-labeler/SKILL.md
as the canonical source of truth. The workflow prompt now references the
skill file instead of inlining the rules, keeping the workflow focused
on workflow-specific concerns (guardrails, target resolution, output).

This enables:
- Batch accuracy evaluation against the same rules
- Interactive Copilot CLI labeling via the skill
- Future eval scripts in skills/agentic-labeler/scripts/

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown
Contributor

🔍 Skill Validation Results

✅ Static Checks Passed

Skills checked: 16 | Agents checked: 3

Full validator output
Found 1 skill(s)
[agentic-labeler] 📊 agentic-labeler: 1,634 BPE tokens [chars/4: 1,590] (detailed ✓), 6 sections, 0 code blocks
[agentic-labeler]    ⚠  No code blocks — agents perform better with concrete snippets and commands.
[agentic-labeler]    ⚠  No numbered workflow steps — agents follow sequenced procedures more reliably.
✅ All checks passed (1 skill(s))
Found 3 agent(s)
Validated 3 agent(s)

✅ All checks passed (3 agent(s))

⏭️ LLM Evaluation: Skipped

No changed skills with eval tests found.

🔍 Full results and investigation steps

@PureWeen PureWeen merged commit 7fd2ca0 into main May 15, 2026
18 of 19 checks passed
@PureWeen PureWeen deleted the feature/agentic-labeler branch May 15, 2026 15:34
@github-actions github-actions Bot added this to the .NET 10.0 SR8 milestone May 15, 2026
SyedAbdulAzeemSF4852 pushed a commit to SyedAbdulAzeemSF4852/maui that referenced this pull request May 19, 2026
### Context

Add ability for maintainers to trigger the AzDO PR review pipeline via
`/review` comment on PR

### Notes

- The workflow allways runs from main - so users cannot chage behavior
in their PRs
- Unprivileged users slash command is ignored
- The 'agentic-labeler.md‎' pipeline referenced in comments of this
pipeline is being added by dotnet#35382

### Tested execution:

- GitHub Actions run:
https://github.com/dotnet/maui/actions/runs/25163585137

- DevDiv pipeline run:
https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=13980704

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
SyedAbdulAzeemSF4852 pushed a commit to SyedAbdulAzeemSF4852/maui that referenced this pull request May 19, 2026
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

## Description

Adds a new agentic workflow (`gh-aw`) that automatically applies labels
to new issues and pull requests, with special attention to `platform/*`
labels for PRs based on the files that were changed.

Inspired by [githubnext/agentics
issue-triage](https://github.com/githubnext/agentics/blob/main/workflows/issue-triage.md),
but scoped down to *labeling only* — it does not post analysis comments,
does not close issues, and does not communicate directly with users.

## Triggers

- `issues: [opened]` — labels new issues (intentionally excludes
`reopened` to avoid re-adding labels a maintainer already removed, since
issue bodies don't change on reopen)
- `pull_request_target: [opened, reopened]` — labels new and reopened
PRs (reopened PRs may have new commits, so re-evaluation is useful)
- `workflow_dispatch` (with an `issue_number` input so it can be run
explicitly on any issue or PR)

## Labeling rules

- Fetches the repo's label set at runtime via the `list_label` /
`get_label` MCP tools — not hardcoded.
- Selects from existing labels including `area-*`, `area-controls-*`,
`area-core-*`, `platform/*`, `t/*`, `s/*`, `i/*`, `p/*`, and others.
- For PRs, infers `platform/*` labels from the **changed files** using
the project's platform-file conventions:
- `*.android.cs`, `/Platform/Android/`, `/Platforms/Android/` →
`platform/android`
- `*.ios.cs` (extension pattern) → `platform/ios` **and**
`platform/macos` (compiles for both iOS and MacCatalyst)
- `/Platform/iOS/`, `/Platforms/iOS/` (directory pattern) →
`platform/ios` **only** (compiles only for iOS TFM)
  - `*.maccatalyst.cs`, `/Platform/MacCatalyst/` → `platform/macos` only
  - `*.windows.cs`, `/Platform/Windows/` → `platform/windows`
  - `*.tizen.cs`, `/Tizen/` → `platform/tizen`
- Conservative by default: if nothing clearly applies, the agent calls
`noop` instead. One `add_labels` call allowed per run (`max: 1`).

## Security model

- **Read-only agent** — permissions are `contents: read`, `issues:
read`, `pull-requests: read`. The agent runs inside a sandboxed
container with no write credentials.
- **Safe-output writes** — label application happens in a separate
safe-output job with write permissions, capped at 1 call.
- **`roles: all`** — allows community contributors' issues/PRs to be
labeled. Safe because the agent is read-only and the only write surface
is `add_labels`.
- **`min-integrity: none`** — allows the MCP gateway to return content
from all authors (including first-time contributors), so the agent can
read the body it needs to label.
- **Prompt-injection guardrails** — explicit instructions telling the
agent to ignore labeling instructions in issue/PR bodies, never use an
`item_number` from untrusted text, and derive labels only from technical
content and file paths.
- **Noise suppression** — `noop`, `missing-tool`, `report-incomplete`,
and `report-failure` are all configured to not create tracker issues.

## Files

- `.github/workflows/agentic-labeler.md` — the agentic workflow source
- `.github/workflows/agentic-labeler.lock.yml` — compiled GitHub Actions
YAML (generated by `gh aw compile`, v0.68.3)

## Notes for reviewers

- This is consistent with the other `gh-aw` workflows in the repo
(`ci-doctor`, `copilot-evaluate-tests`, `daily-repo-status`).
- Draft because we may want to validate behavior on a few real
issues/PRs (via `workflow_dispatch`) before enabling on every new
issue/PR.

---------

Co-authored-by: Shane Neuville <shneuvil@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PureWeen added a commit to PureWeen/skills that referenced this pull request May 20, 2026
…curity

Three adversarial multi-model reviews (Claude Opus 4.7 / Claude Sonnet 4.6 /
GPT-5.3-Codex) of the gh-aw-guide independently identified the systemic root
cause behind the dotnet/maui agentic-labeler 'max: 1 silently drops labels'
bug. This commit applies the consensus fixes.

## 1. Replace generic max: 'calls' framing with per-type unit table

Before:
  'max:' is a blast-radius cap... Set max: to exactly the number of
  intentional calls your orchestration instructions require.

After: an explicit table of what max: counts for each of the 30+
safe-output types. Critically, add-labels / add-reviewer / hide-comment /
create-pull-request-review-comment / reply-to-pull-request-review-comment /
resolve-pull-request-review-thread all count *items* (labels, reviewers,
comments, threads), NOT API calls. The agent batches multiple items per
call, so 'max: 1' on a multi-item type silently drops every item past the
first. This is exactly what shipped in dotnet/maui#35382 and was fixed by
dotnet/maui#35540.

## 2. Add new 'Add Labels — Security Hardening' section

The official add-labels reference exposes 'allowed:' (glob allow-list) and
'blocked:' (glob deny-list) as infrastructure-level filters. Neither was
mentioned anywhere in SKILL.md before this commit. Workflows with
'roles: all' that omit 'allowed:' let a prompt-injected agent apply any
label in the repo, including labels that drive downstream automation
('approved-for-merge', 'needs-backport', label_command: triggers).

## 3. Soften the min-integrity defaults overstatement

Before: 'Public repos default to min-integrity: approved when
unconfigured. Private repos default to min-integrity: none.'

After: Documents that the default is computed at runtime by
'determine-automatic-lockdown' based on event / actor / repo context — not
a static per-visibility rule. Adds a prominent ⚠️ callout that private
repos resolving to 'none' is dangerous when the workflow has
write-capable safe-outputs, because 'private repo = trusted' is a
common-but-wrong assumption.

## 4. Fix the Copilot max-turns example

Before: 'engine: { id: copilot, max-turns: 15 }'

The engine feature-comparison table immediately below this line shows
Copilot as ✗ for max-turns and Claude as ✓ — the example contradicted
the matrix. Changed the example to claude and added an inline note that
Copilot uses max-continuations instead.

## Findings reviewed but discarded after upstream verification

- 'noop: report-as-issue: false' is real (documented under
  experimental/monitoring-with-projects/), not hallucinated as one agent
  claimed.
- 'merge-pull-request' is a real safe-output (ADR-27193 dated
  2026-04-19, implemented in pkg/workflow/merge_pull_request.go), not
  fabricated as the doc-drift agent claimed.
- The 'Do NOT set min-integrity explicitly (compiler bug)' line at
  architecture.md:214 — already fixed in PureWeen/skills main; the
  reviewing agents saw a stale local copy.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-tooling XAML & C# Hot Reload, XAML Editor, Live Visual Tree, Live Preview, Debugging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants