Agentic-labeler: route AI-agent and platform labels correctly

[PureWeen]

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Root Cause

The labeler treated agent skills and AI-assisted workflows as generic infrastructure, so it never selected the more-specific area-ai-agents label. Its issue-platform guidance also inferred labels too broadly and included the unsupported platform/tizen label.

Description of Change

• Routes Copilot CLI agents, agent skills, agentic workflows, evaluations, and AI-assisted development to area-ai-agents.
• Keeps area-infrastructure for generic CI execution, authentication, scheduling, dependency flow, and pipeline plumbing, with an explicit dominant-subject tie-break.
• Distinguishes setup concerns such as workload availability and target-framework recognition (area-setup) from debugging, editor, build-task, and tooling behavior (area-tooling).
• Never applies platform/tizen, while still selecting the appropriate area-* label for Tizen-related content.
• Infers issue platform labels only from explicitly named affected platforms; generic claims such as "all platforms" do not infer labels, while an explicit affected-platform list takes precedence.
• Expands the active Vally suite from 21 to 25 scenarios, including deterministic regression coverage for area-ai-agents, platform exclusion, and exact area selection.
• Explicitly activates agentic-labeler in every capability scenario and requires labels-only output so the suite measures skill behavior rather than nondeterministic skill discovery or explanatory prose.

Validation

• Rebased onto main with @microsoft/vally-cli@0.10.0 pinned by the Skill Validation workflow.
• Full three-trial agentic-labeler Vally 0.10.0 evaluation passed at 99.8% (threshold: 85.0%).
• JUnit aggregate: 0 failures, 0 errors.
• Strict Vally 0.10.0 static validation passed.
• Harness hermeticity gate passed.
• Skill Validation run: https://github.com/dotnet/maui/actions/runs/29609999793

Issues Fixed

Follow-up to and replacement for #35570.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36639

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36639"

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This pull request updates the agentic-labeler skill spec and its Vally evaluation suite to refine platform inference for issues and to explicitly exclude platform/tizen, while adding new evaluation scenarios to lock in the intended tie-breaking behavior.

Changes:

• Updated agentic-labeler rules to never apply platform/tizen, including when Tizen is mentioned in issues or Tizen-specific files are changed.
• Refined issue platform inference to require explicitly named affected platforms (generic “all platforms” claims alone do not infer platform labels).
• Expanded the Vally eval suite with 4 new issue scenarios covering control-vs-generic area tie-breaks, explicit platform subsets, generic platform claims, and Tizen exclusion.

Show a summary per file

File	Description
.github/skills/agentic-labeler/SKILL.md	Updates the labeling rules to exclude platform/tizen and clarifies issue platform inference expectations.
.github/skills/agentic-labeler/tests/eval.vally.yaml	Bumps suite version and adds 4 new issue stimuli to validate “exactly one area” + platform inference rules (including Tizen exclusion).

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 0

[github-actions]

Skill Validation Results

@PureWeen — new skill validation results are available based on this last commit: 2ba6eb8.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

✅ Skill Validation Results — 2ba6eb8 · Agentic-labeler: route AI-agent and platform labels correctly · 2026-07-17T20:13:16Z

✅ Static Checks Passed

Skills: 21 | Eval specs linted: 10

Full lint output

── .github/skills/agentic-labeler/tests/eval.vally.yaml
npm warn deprecated prebuild-install@7.1.3: No longer maintained. Please contact the author of the relevant native addon; alternatives are available.
✔ .github/skills/agentic-labeler/tests/eval.vally.yaml is valid
── .github/skills/analyze-sessions/tests/eval.vally.yaml
✔ .github/skills/analyze-sessions/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/eval.capability.vally.yaml
✔ .github/skills/code-review/tests/eval.capability.vally.yaml is valid
── .github/skills/code-review/tests/eval.inline-findings.vally.yaml
✔ .github/skills/code-review/tests/eval.inline-findings.vally.yaml is valid
── .github/skills/code-review/tests/eval.vally.yaml
✔ .github/skills/code-review/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/hermeticity.vally.yaml
✔ .github/skills/code-review/tests/hermeticity.vally.yaml is valid
── .github/skills/evaluate-pr-tests/tests/eval.vally.yaml
✔ .github/skills/evaluate-pr-tests/tests/eval.vally.yaml is valid
── .github/skills/pr-review/tests/eval.gh-auth.vally.yaml
✔ .github/skills/pr-review/tests/eval.gh-auth.vally.yaml is valid
── .github/skills/try-fix/tests/eval.vally.yaml
✔ .github/skills/try-fix/tests/eval.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml is valid

✅ LLM Evaluation Passed

1/1 eval suite(s) met threshold

Suite	Score	Threshold	Verdict
agentic-labeler-capabilities	1.00	0.85	✅

Harness hermeticity (negative control)

✅ Hermetic — the negative-control stimulus correctly came back unauthenticated (anonymous core rate limit; no GitHub token leaked into the agent env).

📊 agentic-labeler — eval report

Eval Results

Timestamp: 2026-07-17T20:13:06.844Z

Capability suite for the agentic-labeler skill — verifies it derives the correct 'area-' and 'platform/' labels from changed-file path conventions (and explicit platform mentions on issues), applies the iOS/MacCatalyst extension-vs-directory distinction, prefers area-infrastructure for generic CI infrastructure, noops automated-merge and already-labeled dependency PRs, resists label instructions injected into issue bodies, prefers a specific area over its parent, routes agent skills and AI-assisted development to area-ai-agents, distinguishes explicit affected-platform lists from generic platform claims, excludes Tizen platform labels, and never applies out-of-scope (t/* i/* s/* p/* partner/* perf/*) labels.

Stimulus	Skills	Graders	Pass Rate	pass@k	pass^k	Duration (median)	Tokens (median)	Turns (median)	Tool Calls (median)	Verdict
agent-skill-area	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	17.3s	44,571	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
ai-review-workflow-area	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	17.9s	44,564	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
android-extension-and-area-essentials	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	18.7s	45,205	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
ci-workflow-infrastructure	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	13.4s	44,470	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
handlers-android-subdir	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	14.0s	44,625	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
ios-directory-collectionview	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	20.7s	44,823	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
ios-directory-only-not-macos	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	20.0s	44,866	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
ios-extension-dual-platform	agentic-labeler (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	19.2s	44,869	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
issue-dnceng-codeflow-infrastructure	agentic-labeler (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	14.2s	44,569	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
issue-explicit-platforms-no-triage	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	12.9s	44,609	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
issue-generic-platform-claim-no-inference	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	15.3s	44,581	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
issue-prompt-injection-resistance	agentic-labeler (2×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	14.8s	44,686	2	1 calls (median) total across 3 trials: skill: 2</details>	✅
issue-safearea-named-platform-subset	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	16.4s	44,616	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
issue-tabbedpage-over-navigation	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	13.5s	44,548	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
issue-tizen-platform-excluded	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	15.4s	44,543	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
maccatalyst-only-not-ios	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	17.6s	44,767	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
maps-exact-label-name	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	21.7s	44,821	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
multi-platform-applies-all	agentic-labeler (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	16.2s	44,825	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
noop-automated-merge-pr	agentic-labeler (3×)	✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	12.2s	44,551	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
noop-dependency-bump	agentic-labeler (3×)	✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	11.8s	44,561	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
pr-no-triage-labels	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	16.6s	44,696	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
revert-android-collectionview-scope	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	19.4s	44,779	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
shell-area-no-platform	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	14.4s	44,588	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
windows-collectionview	agentic-labeler (3×)	✅ output-contains 3/3 ✅ output-not-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	14.9s	44,635	2	1 calls (median) total across 3 trials: skill: 3</details>	✅
xaml-source-generator-area	agentic-labeler (3×)	✅ output-contains 3/3 ✅ prompt 3/3	3/3	100.0%	100.0%	13.2s	44,690	2	1 calls (median) total across 3 trials: skill: 3</details>	✅

Model: claude-opus-4.6 | Judge: claude-opus-4.6 | Executor: copilot-sdk

🔍 Full results and investigation steps

[Copilot]

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 0 new

[kubaflo]

🔍 AI-generated review (multi-model pipeline: Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), on behalf of @kubaflo.

Multi-model review — PR #36639 · agentic-labeler: Tizen exclusion + exactly-one-area scenarios

Reviewed both files at head d7762c91: .github/skills/agentic-labeler/SKILL.md and .github/skills/agentic-labeler/tests/eval.vally.yaml (+162/−7). This supersedes the closed #35570.

Independent assessment

SKILL.md gains a hard platform/tizen is never applied rule (removes the tizen row from the file→label table; Tizen files still get a normal area-*, just no platform/*) and a rewritten issue platform-inference section (only explicitly-named affected platforms get labels; a named list beats a generic "all platforms" phrase; incidental / "not affected" / label-request mentions are ignored). The eval fixture bumps 1.0.0→1.1.0 and adds scenarios 22–25.

Ground-truth verified — I cross-checked every new scenario against the real maintainer labels on its cited source issue, and against the SKILL.md tie-break rules:

Scenario	Asserts	Real issue labels	SKILL.md rule
22 (#35490)	area-controls-tabbedpage, platform/ios, ¬area-navigation	area-controls-tabbedpage + area-navigation + platform/ios	"specific control beats generic" (line 72) ✓
23 (#35501)	area-safearea, platform/ios+platform/android, ¬platform/windows	area-safearea + area-layout + all 4 platforms	"sub-area beats parent" (line 73) + named-list-wins ✓
24	area-safearea, no platform/*	—	generic "all platforms" → no platform label ✓
25 (#31806)	area-tooling, no platform/*	area-tooling + platform/tizen	area-tooling = workload/TFM surface (line 56); Tizen excluded (line 101) ✓

All three area labels (area-controls-tabbedpage, area-safearea, area-tooling) exist in dotnet/maui, and area-tooling (not area-infrastructure, which is CI/agent-infra-only) is the correct call for a workload/SDK target-framework issue — confirmed by #31806's real labels. The YAML parses cleanly, 25 stimuli, no duplicate names, schema matches the existing 21, and the "21 scenarios"→"scenarios" header edit leaves no stale count.

Findings — both non-blocking

💡 Suggestion — scenarios 24 & 25 use a bare output-not-contains: "platform/" grader (Opus)

Every other not-contains grader in the suite (~20) asserts a fully-qualified label (platform/windows, area-navigation, …). Scenarios 24 & 25 alone assert the bare prefix platform/. The intended pass answers (area-safearea, area-tooling) don't contain that substring, so the happy path is safe — but if the agent emits any explanatory text containing platform/, the floor false-fails. This is sharpest for scenario 25: the behavior under test is recognizing Tizen is excluded, which naturally invites the agent to name platform/tizen as the dropped label — tripping the grader. The Return labels only instruction mitigates it. Consider asserting the specific label that must be absent (e.g. platform/tizen for #25) to match the rest of the suite.

💡 Suggestion — top-level "One or more platform/*" cardinality (GPT-5.5)

SKILL.md line 2 still reads "One or more platform/*", but the canonical behavior (and new scenarios 24/25) allow zero platform labels. Since this line was touched to append the Tizen exception, "Zero or more platform/*" would read more precisely. Minor — the detailed rules already make zero-platform explicit.

Model verdicts

Model	Verdict
Opus 4.8	LGTM (+ grader-prefix suggestion)
Gemini 3.1 Pro	LGTM (no findings)
GPT-5.5	NEEDS_CHANGES (cardinality wording)

Verdict: 🟢 LGTM — confidence: high

No defects. The new scenarios are precise, YAML-valid, and impressively well-grounded in the real maintainer labels of their source issues, matching the rewritten SKILL.md rules verbatim. The two suggestions above are optional polish — the platform/tizen grader-prefix on scenario 25 is the more worthwhile one to tighten.

_{ℹ Automated multi-model review (Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro). Advisory — a human maintainer makes the final call.}

[Copilot]

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 3

[Copilot]

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 1

[kubaflo]

🔍 AI-generated review (multi-model pipeline: Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), on behalf of @kubaflo — round 2 at 0d39e182.

Multi-model review — PR #36639 (agentic-labeler: AI-agent + platform routing)

Successor to the closed #35570 (Tizen-exclusion + affected-platform work, already reviewed there). This round's new material — a area-ai-agents routing taxonomy — plus the just-pushed "Stabilize agentic labeler capability evals" (0d39e182), which lands cleanly.

New taxonomy — verified sound

Check	Result
area-ai-agents label exists in-repo	✅ real target
Split enforced both ways	✅ scenarios #19 ai-review-workflow-area / #20 agent-skill-area assert area-ai-agents and output-not-contains: area-infrastructure
Tie-break unambiguous	✅ 0d39e182 clarifies command-parsing/trigger-dispatch for AI-review = agent behavior; generic CI plumbing = infrastructure
YAML parse @Head	✅ version: 1.3.0, 25 stimuli
Tizen/affected-platform scenarios	✅ same as #35570 (labels all real)

The stabilization commit resolves my round-1/round-2 findings

0d39e182 adds a per-stimulus output constraint to all 25 scenarios ("Return only the labels to apply, one per line, or NONE; do not explain or mention excluded labels"). This directly hardens the previously-brittle output-not-contains: "platform/" / output-not-contains: area-infrastructure graders — a model can no longer trip them by mentioning an excluded label in prose. It also fixes the stale "CI/agent-infra files" wording in the results template. Net effect on CI:

• Earlier attempt (at the prior head) flaked Failed → Passed; at 0d39e182 the suite is clean green: agentic-labeler-capabilities 0.97 / 0.85 ✅, overall Static Passed · LLM Passed, aggregate "Skill validation passed". The instability I was about to flag is addressed at the source.

Findings

• 💡 (non-blocking) Dogfooding. This PR is labeled area-infrastructure; under its own new rule (.github/skills/** → area-ai-agents) the post-merge labeler would route it to area-ai-agents. Harmless (label predates the rule) — a nice live confirmation the taxonomy behaves as intended.

Verdict: ✅ Approve · confidence high

The area-ai-agents taxonomy is correct (real label, symmetric enforcement, unambiguous tie-break), the Tizen/affected-platform work was already vetted on #35570, and the stabilization commit turns the previously-flaky eval into a clean green (0.97). No residual blocking or warning-level findings. LGTM.

_{ℹ Automated multi-model review (Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), cross-pollinated and verified against the files + live CI at 0d39e182. Advisory — a human maintainer makes the final merge decision.}

[Copilot]

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 0 new

[kubaflo]

🔍 AI-generated review (multi-model pipeline: Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), on behalf of @kubaflo — round 3 at 2ba6eb86.

Multi-model review — PR #36639 (agentic-labeler) — re-approve at new head

Re-review after 2ba6eb86 "Fix remaining labeler eval regressions" landed on top of my round-2 approve (0d39e182). This commit is a strict improvement and, notably, resolves the last standing 💡 from earlier rounds.

What 2ba6eb86 changes

• Precise tooling/setup/templates routing (SKILL.md). The previously-fuzzy "Hot reload, build, MSBuild, workload, project templates, tooling → area-tooling/area-templates/area-setup as appropriate" is split into deterministic per-surface rules:
  • Installation, workload availability, target-framework recognition, requirements, platform support → area-setup
  • Hot reload, debugging, editor experiences, build tasks, MSBuild/tooling behavior → area-tooling
  • Project templates → area-templates
    This directly addresses the round-1/round-2 note that scenario-25's hard area-tooling requirement was self-fragile against the "as appropriate" wording.
• eval.vally.yaml → v1.4.0. Scenario-25 now asserts area-setup (the workload/SDK target-framework setup surface), consistent with the refined routing, and three more scenarios gain the output-not-contains: "platform/" guard.

CI — clean green at head

evaluate (agentic-labeler) success, hermeticity success, static success, aggregate "Skill validation passed", results comment Overall Passed · LLM Passed at 2ba6eb8.

Verdict: ✅ Approve (re-affirmed at 2ba6eb86) · confidence high

Everything from the round-2 approval still holds (sound area-ai-agents taxonomy, symmetric enforcement, unambiguous tie-break), and this commit resolves the one remaining suggestion while keeping the suite green. No residual findings. LGTM.

_{ℹ Automated multi-model review (Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro), cross-pollinated and verified against the files + live CI at 2ba6eb86. Advisory — a human maintainer makes the final merge decision.}