diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 90dad179ff44..c97b4299a8fb 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -245,7 +245,18 @@ Skills are modular capabilities that can be invoked directly or used by agents. #### User-Facing Skills -1. **issue-triage** (`.github/skills/issue-triage/SKILL.md`) +1. **pr-review** (`.github/skills/pr-review/SKILL.md`) + - **Purpose**: End-to-end PR review orchestrator — 3 phases: pr-preflight, try-fix, pr-report. Gate runs separately before this skill via Review-PR.ps1. + - **Trigger phrases**: "review PR #XXXXX", "work on PR #XXXXX", "fix issue #XXXXX", "continue PR #XXXXX" + - **Capabilities**: Multi-model fix exploration, alternative comparison, PR review recommendation + - **Do NOT use for**: Just running tests manually → Use `sandbox-agent` + - **Phase instructions** (in `.github/pr-review/`): + - `pr-preflight.md` — Context gathering from issue/PR + - `pr-report.md` — Final recommendation + - **Phase skill**: `try-fix` — Multi-model fix exploration + - **Note**: Gate (test verification) runs as a script step in `Review-PR.ps1` before this skill is invoked. Gate result is passed in the prompt. + +2. **issue-triage** (`.github/skills/issue-triage/SKILL.md`) - **Purpose**: Query and triage open issues that need milestones, labels, or investigation - **Trigger phrases**: "find issues to triage", "show me old Android issues", "what issues need attention" - **Scripts**: `init-triage-session.ps1`, `query-issues.ps1`, `record-triage.ps1` @@ -286,8 +297,8 @@ Skills are modular capabilities that can be invoked directly or used by agents. - **Trigger phrases**: "write XAML tests for #XXXXX", "test XamlC behavior", "reproduce XAML parsing bug" - **Output**: Test files for Controls.Xaml.UnitTests -8. **verify-tests-fail-without-fix** (`.github/skills/verify-tests-fail-without-fix/SKILL.md`) - - **Purpose**: Verifies UI tests catch the bug before fix and pass with fix +9. **verify-tests-fail-without-fix** (`.github/skills/verify-tests-fail-without-fix/SKILL.md`) + - **Purpose**: Verifies tests catch the bug before fix and pass with fix. Auto-detects test type (UI, device, unit, XAML) and dispatches to the appropriate runner. - **Two modes**: Verify failure only (test creation) or full verification (test + fix) - **Used by**: After creating tests, before considering PR complete diff --git a/.github/pr-review/pr-gate.md b/.github/pr-review/pr-gate.md index 817496178333..78f98ddc5d54 100644 --- a/.github/pr-review/pr-gate.md +++ b/.github/pr-review/pr-gate.md @@ -1,8 +1,9 @@ -# PR Gate — Test Verification +# PR Gate - Test Before and After Fix > **⛔ This phase MUST pass before continuing to Try-Fix. If it fails, stop and inform user.** -> 🚨 Gate verification MUST run via task agent — never inline. +> In CI (Review-PR.ps1), the gate runs `verify-tests-fail.ps1` directly as a script step. +> For manual usage, you can invoke it yourself or via a task agent. --- @@ -26,41 +27,32 @@ Choose a platform that is BOTH affected by the bug AND available on the current ## Steps -1. **Check if tests exist:** +1. **Detect tests in PR** using the shared detection script: ```bash - gh pr view XXXXX --json files --jq '.files[].path' | grep -E "TestCases\.(HostApp|Shared\.Tests)" + pwsh .github/scripts/shared/Detect-TestsInDiff.ps1 -PRNumber XXXXX ``` - If NO tests exist → inform user, suggest `write-tests-agent`. Gate is ⚠️ SKIPPED. + This auto-detects all test types: UI tests, device tests, unit tests, XAML tests. + If NO tests detected → inform user, suggest `write-tests-agent`. Gate is ⚠️ SKIPPED. -2. **Select platform** — must be affected by bug AND available on host (see Platform Selection above). +2. **Select platform** — must be affected by bug AND available on host (see table above). -3. **Run verification via task agent** (MUST use task agent — never inline): +3. **Run verification** via `verify-tests-fail.ps1`: + ```bash + pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 \ + -Platform {platform} -RequireFullVerification + ``` + In CI, `Review-PR.ps1` calls this script directly. For manual usage, you can also invoke + it via a task agent for isolation: ``` Invoke the `task` agent with this prompt: "Invoke the verify-tests-fail-without-fix skill for this PR: - Platform: {platform} - - TestFilter: 'IssueXXXXX' - RequireFullVerification: true Report back: Did tests FAIL without fix? Did tests PASS with fix? Final status?" ``` -**Why task agent?** Running inline allows substituting commands and fabricating results. Task agent runs in isolation. - ---- - -## Expected Result - -``` -╔═══════════════════════════════════════════════════════════╗ -║ VERIFICATION PASSED ✅ ║ -╠═══════════════════════════════════════════════════════════╣ -║ - FAIL without fix (as expected) ║ -║ - PASS with fix (as expected) ║ -╚═══════════════════════════════════════════════════════════╝ -``` - --- ## If Gate Fails @@ -72,25 +64,44 @@ Choose a platform that is BOTH affected by the bug AND available on the current ## Output File +> 🚨 **CRITICAL OUTPUT RULES:** +> - Write gate results ONLY to `gate/content.md` — NEVER copy gate results into other phases (pre-flight, try-fix, report) +> - Use the EXACT template below — no extra explanations, no "Reason:" paragraphs, no "Notes:" sections +> - Keep it SHORT — the template is the complete output + ```bash mkdir -p CustomAgentLogsTmp/PRState/{PRNumber}/PRAgent/gate ``` -Write `content.md`: +Write `content.md` using this **exact** template (fill in values, don't add anything else): + ```markdown ### Gate Result: {✅ PASSED / ❌ FAILED / ⚠️ SKIPPED} **Platform:** {platform} -**Mode:** Full Verification -- Tests FAIL without fix: {✅/❌} -- Tests PASS with fix: {✅/❌} +| # | Type | Test Name | Filter | +|---|------|-----------|--------| +| 1 | {type} | {name} | `{filter}` | + +| Step | Expected | Actual | Result | +|------|----------|--------|--------| +| Without fix | FAIL | {FAIL/PASS} | {✅/❌} | +| With fix | PASS | {FAIL/PASS} | {✅/❌} | +``` + +If gate is SKIPPED (no tests found), write only: + +```markdown +### Gate Result: ⚠️ SKIPPED + +No tests detected in PR. Suggest adding tests via `write-tests-agent`. ``` --- ## Common Mistakes -- ❌ Running inline — MUST use task agent -- ❌ Using `BuildAndRunHostApp.ps1` — that runs ONE direction; the skill does TWO -- ❌ Claiming results from a single test run — script does TWO runs automatically +- ❌ Adding verbose explanations to gate/content.md — use the exact template above +- ❌ Copying gate results into try-fix/content.md or report/content.md — gate results belong ONLY in gate/content.md +- ❌ Skipping gate because tests are device tests, not UI tests — the skill supports all test types diff --git a/.github/pr-review/pr-report.md b/.github/pr-review/pr-report.md index ffe233ad1172..89651509fd3f 100644 --- a/.github/pr-review/pr-report.md +++ b/.github/pr-review/pr-report.md @@ -4,11 +4,14 @@ > 🚨 **DO NOT post any comments.** This phase only produces output files. +> 🚨 **DO NOT duplicate content from other phases.** Reference gate/try-fix results by status only (e.g., "Gate: ✅ PASSED") — do NOT copy their full output into report/content.md. + --- ## Prerequisites -- Phases 1-3 (Pre-Flight, Gate, Try-Fix) must be complete before starting +- Phases 1-2 (Pre-Flight, Try-Fix) must be complete before starting +- Gate result is available from the prompt (ran separately before this skill) --- diff --git a/.github/scripts/BuildAndRunHostApp.ps1 b/.github/scripts/BuildAndRunHostApp.ps1 index 70f92a297f9a..76959772f48e 100644 --- a/.github/scripts/BuildAndRunHostApp.ps1 +++ b/.github/scripts/BuildAndRunHostApp.ps1 @@ -313,8 +313,9 @@ try { # Save test output to file $testOutput | Out-File -FilePath $testOutputFile -Encoding UTF8 - # Display test output - $testOutput | ForEach-Object { Write-Host $_ } + # Output test results to the output stream so callers can capture them + # (Write-Host goes to the Information stream which is not captured by 2>&1) + $testOutput | ForEach-Object { Write-Output $_ } $testExitCode = $LASTEXITCODE diff --git a/.github/scripts/EstablishBrokenBaseline.ps1 b/.github/scripts/EstablishBrokenBaseline.ps1 index 2a6d57506ec2..0fe68a2f8d34 100644 --- a/.github/scripts/EstablishBrokenBaseline.ps1 +++ b/.github/scripts/EstablishBrokenBaseline.ps1 @@ -64,6 +64,9 @@ $script:TestPathPatterns = @( "*.Tests/*", "*.UnitTests/*", "*TestCases*", + "*TestUtils*", + "*DeviceTests.Runners*", + "*DeviceTests.Shared*", "*snapshots*", "*.png", "*.jpg", diff --git a/.github/scripts/Review-PR.ps1 b/.github/scripts/Review-PR.ps1 index d13e2929ab5f..a88a9379cdd9 100644 --- a/.github/scripts/Review-PR.ps1 +++ b/.github/scripts/Review-PR.ps1 @@ -3,12 +3,12 @@ Runs a PR review using Copilot CLI with skill-based prompts. .DESCRIPTION - Orchestrates a 5-step PR review by invoking Copilot CLI with skill prompts: + Orchestrates a PR review by invoking scripts and Copilot CLI: Step 0: Branch setup - Create review branch from main, merge PR squashed - Step 1: pr-review skill - 4-phase review (Pre-Flight, Gate, Try-Fix, Report) - Step 2: pr-finalize skill - Verify PR title/description match implementation - Step 3: Post AI Summary - Directly runs posting scripts (review + finalize) + Step 1: Gate - Run test verification directly (verify-tests-fail.ps1) + Step 2: pr-review skill - 3-phase review (Pre-Flight, Try-Fix, Report) + Step 3: Post AI Summary - Directly runs posting scripts Step 4: Apply labels - Apply agent labels based on review results By default, the script checks out main and creates a review branch from it. @@ -438,33 +438,135 @@ function Invoke-CopilotStep { } # ═════════════════════════════════════════════════════════════════════════════ -# STEP 1: PR Review (4-phase skill) +# STEP 1: Gate - Test Before and After Fix (script, no copilot agent) # ═════════════════════════════════════════════════════════════════════════════ -$step1Prompt = @" -Use a skill to review PR #$PRNumber. +Write-Host "" +Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Yellow +Write-Host "║ STEP 1: GATE — TEST VERIFICATION ║" -ForegroundColor Yellow +Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Yellow -$platformInstruction -$autonomousRules +$gateOutputDir = Join-Path $RepoRoot "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/gate" +New-Item -ItemType Directory -Force -Path $gateOutputDir | Out-Null -📁 Write phase output to ``CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/{phase}/content.md`` +# Detect tests in PR +Write-Host " 🔍 Detecting tests in PR #$PRNumber..." -ForegroundColor Cyan +$detectScript = Join-Path $PSScriptRoot "shared/Detect-TestsInDiff.ps1" +& pwsh -NoProfile -Command "& '$detectScript' -PRNumber $PRNumber" 2>&1 | ForEach-Object { Write-Host " $_" } + +# Determine platform for gate +$gatePlatform = if ($Platform) { $Platform } else { "android" } +Write-Host " 🧪 Running gate on platform: $gatePlatform" -ForegroundColor Cyan + +$verifyScript = Join-Path $PSScriptRoot "../skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1" +& pwsh -NoProfile -File "$verifyScript" -Platform $gatePlatform -PRNumber $PRNumber -RequireFullVerification 2>&1 | ForEach-Object { Write-Host " $_" } +$gateExitCode = $LASTEXITCODE + +# Exit code: 0 = passed, 1 = verification failed, 2 = no tests detected +$gateResult = switch ($gateExitCode) { + 0 { "PASSED" } + 2 { "SKIPPED" } + default { "FAILED" } +} +$gateColor = switch ($gateResult) { "PASSED" { "Green" } "SKIPPED" { "Yellow" } default { "Red" } } +Write-Host " 📁 Gate result: $gateResult" -ForegroundColor $gateColor + +# Copy the verification report to gate/content.md if it exists +$verificationReport = Join-Path $gateOutputDir "verify-tests-fail/verification-report.md" +if (Test-Path $verificationReport) { + Copy-Item $verificationReport (Join-Path $gateOutputDir "content.md") -Force +} elseif (-not (Test-Path (Join-Path $gateOutputDir "content.md"))) { + # Create gate content based on result + if ($gateResult -eq "SKIPPED") { + $skipContent = @" +### Gate Result: ⚠️ SKIPPED + +No tests were detected in this PR. + +**Recommendation:** Add tests to verify the fix using the ``write-tests-agent``: + +`````` +@copilot write tests for this PR +`````` + +The agent will analyze the issue, determine the appropriate test type (UI test, device test, unit test, or XAML test), and create tests that verify the fix. "@ + $skipContent | Set-Content (Join-Path $gateOutputDir "content.md") + } else { + "### Gate Result: $(if ($gateExitCode -eq 0) { '✅ PASSED' } else { '❌ FAILED' })`n`n**Platform:** $gatePlatform" | + Set-Content (Join-Path $gateOutputDir "content.md") + } +} -Invoke-CopilotStep -StepName "STEP 1: PR REVIEW" -Prompt $step1Prompt | Out-Null +# Post gate result as a separate PR comment +$postGateScript = Join-Path $PSScriptRoot "post-gate-comment.ps1" +if (Test-Path $postGateScript) { + try { + if ($DryRun) { + & $postGateScript -PRNumber $PRNumber -DryRun + } else { + & $postGateScript -PRNumber $PRNumber + } + } catch { + Write-Host " ⚠️ Failed to post gate comment (non-fatal): $_" -ForegroundColor Yellow + } +} else { + Write-Host " ⚠️ post-gate-comment.ps1 not found" -ForegroundColor Yellow +} -# Restore review branch — the Copilot agent may have switched branches (e.g. via gh pr checkout) +# Apply gate result label +$gatePassLabel = "s/agent-gate-passed" +$gateFaillabel = "s/agent-gate-failed" +$gateSkipLabel = "s/agent-gate-skipped" +$allGateLabels = @($gatePassLabel, $gateFaillabel, $gateSkipLabel) + +$addLabel = switch ($gateResult) { + "PASSED" { $gatePassLabel } + "SKIPPED" { $gateSkipLabel } + default { $gateFaillabel } +} +$removeLabels = $allGateLabels | Where-Object { $_ -ne $addLabel } + +if (-not $DryRun) { + foreach ($lbl in $removeLabels) { + gh pr edit $PRNumber --remove-label $lbl --repo dotnet/maui 2>$null | Out-Null + } + gh pr edit $PRNumber --add-label $addLabel --repo dotnet/maui 2>$null | Out-Null + if ($LASTEXITCODE -eq 0) { + Write-Host " 🏷️ Label: $addLabel" -ForegroundColor Cyan + } else { + Write-Host " ⚠️ Failed to apply label $addLabel" -ForegroundColor Yellow + } +} else { + Write-Host " [DRY RUN] Would set label: $addLabel" -ForegroundColor Magenta +} + +# Restore review branch git checkout $reviewBranch 2>$null | Out-Null # ═════════════════════════════════════════════════════════════════════════════ -# STEP 2: PR Finalize +# STEP 2: PR Review (3-phase skill: Pre-Flight, Try-Fix, Report) # ═════════════════════════════════════════════════════════════════════════════ +$gateStatusForPrompt = switch ($gateResult) { + "PASSED" { "Gate ✅ PASSED — tests FAIL without fix, PASS with fix." } + "SKIPPED" { "Gate ⚠️ SKIPPED — no tests detected in this PR. Consider suggesting the author add tests." } + default { "Gate ❌ FAILED — tests did NOT behave as expected." } +} + $step2Prompt = @" -Use a skill to finalize PR #$PRNumber. Write findings to ``CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/pr-finalize/pr-finalize-summary.md``. +Use a skill to review PR #$PRNumber. + +$platformInstruction $autonomousRules + +**Gate result (already completed in a prior step):** $gateStatusForPrompt +Do NOT re-run gate verification. The gate phase is handled separately. + +📁 Write phase output to ``CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/{phase}/content.md`` "@ -Invoke-CopilotStep -StepName "STEP 2: PR FINALIZE" -Prompt $step2Prompt | Out-Null +Invoke-CopilotStep -StepName "STEP 2: PR REVIEW" -Prompt $step2Prompt | Out-Null # Restore review branch — the Copilot agent may have switched branches (e.g. via gh pr checkout) git checkout $reviewBranch 2>$null | Out-Null @@ -479,15 +581,18 @@ Write-Host "║ STEP 3: POST AI SUMMARY ║" - Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Magenta $summaryScriptsDir = Join-Path $RepoRoot ".github/scripts" -$dryRunFlag = if ($DryRun) { @('-DryRun') } else { @() } -# 3a: Post PR review phases (pre-flight, gate, try-fix, report) +# Post PR review phases (pre-flight, try-fix, report) $aiSummaryCommentId = $null $reviewScript = Join-Path $summaryScriptsDir "post-ai-summary-comment.ps1" if (Test-Path $reviewScript) { try { Write-Host " 📝 Posting PR review summary..." -ForegroundColor Cyan - $reviewOutput = & $reviewScript -PRNumber $PRNumber @dryRunFlag + if ($DryRun) { + $reviewOutput = & $reviewScript -PRNumber $PRNumber -DryRun + } else { + $reviewOutput = & $reviewScript -PRNumber $PRNumber + } # Capture comment ID from script output (format: COMMENT_ID=) $idLine = $reviewOutput | Where-Object { $_ -match '^COMMENT_ID=' } | Select-Object -Last 1 if ($idLine -match '^COMMENT_ID=(\d+)$') { @@ -503,24 +608,6 @@ if (Test-Path $reviewScript) { Write-Host " ⚠️ post-ai-summary-comment.ps1 not found — skipping review summary" -ForegroundColor Yellow } -# 3b: Post PR finalize section (title, description, code review) -$finalizeScript = Join-Path $summaryScriptsDir "post-pr-finalize-comment.ps1" -if (Test-Path $finalizeScript) { - try { - Write-Host " 📝 Posting PR finalize summary..." -ForegroundColor Cyan - $finalizeArgs = @('-PRNumber', $PRNumber) + $dryRunFlag - if ($aiSummaryCommentId) { - $finalizeArgs += @('-ExistingCommentId', $aiSummaryCommentId) - } - & $finalizeScript @finalizeArgs - Write-Host " ✅ PR finalize summary posted" -ForegroundColor Green - } catch { - Write-Host " ⚠️ PR finalize summary posting failed (non-fatal): $_" -ForegroundColor Yellow - } -} else { - Write-Host " ⚠️ post-pr-finalize-comment.ps1 not found — skipping finalize summary" -ForegroundColor Yellow -} - # ═════════════════════════════════════════════════════════════════════════════ # STEP 4: Apply Labels # ═════════════════════════════════════════════════════════════════════════════ diff --git a/.github/scripts/RunTests.ps1 b/.github/scripts/RunTests.ps1 new file mode 100644 index 000000000000..34c99f8720aa --- /dev/null +++ b/.github/scripts/RunTests.ps1 @@ -0,0 +1,625 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Unified test runner for .NET MAUI — run any test type from a single entry point. + +.DESCRIPTION + This script provides a single entry point for running all types of tests in the + .NET MAUI repository: unit tests, device tests, UI tests, and integration tests. + + For unit tests, the script runs dotnet test directly. + For device tests, UI tests, and integration tests, the script delegates to the + appropriate existing scripts (Run-DeviceTests.ps1, BuildAndRunHostApp.ps1, + Run-IntegrationTests.ps1). + +.PARAMETER TestType + The type of tests to run: + - Unit: Unit tests (runs on build machine via dotnet test) + - Device: Device tests (runs on device/emulator via xharness) + - UI: UI tests with Appium (builds HostApp, deploys, runs dotnet test) + - Integration: Integration tests (template builds, workload validation) + +.PARAMETER Project + The project/module to test. Available values depend on TestType: + + Unit tests: Controls, Xaml, BindingSourceGen, SourceGen, ControlsDesign, + Core, Essentials, Graphics, Resizetizer, Compatibility, DualScreen, AI + (omit to run ALL unit test projects) + + Device tests: Controls, Core, Essentials, Graphics, BlazorWebView, AI + +.PARAMETER Platform + Target platform (required for Device and UI tests): + android, ios, catalyst (or maccatalyst), windows + +.PARAMETER TestFilter + Test filter expression passed to dotnet test --filter (e.g., "FullyQualifiedName~Issue12345") + +.PARAMETER Category + Test category filter. Converted to --filter "Category=" for unit/UI tests, + or passed as -Category for integration tests. + Cannot be used together with -TestFilter. + +.PARAMETER Configuration + Build configuration: Debug or Release (default: Debug for Unit/UI, Release for Device) + +.PARAMETER DeviceUdid + Specific device UDID for Device or UI tests (optional — auto-detected if omitted) + +.PARAMETER Rebuild + Force a clean rebuild (no incremental build) + +.PARAMETER List + Show all available test types, projects, and usage examples, then exit. + +.PARAMETER ResultsDirectory + Directory for test results (default: artifacts/test-results) + +.EXAMPLE + ./RunTests.ps1 -List + # Show all available test types and projects + +.EXAMPLE + ./RunTests.ps1 -TestType Unit -Project Controls + # Run Controls unit tests + +.EXAMPLE + ./RunTests.ps1 -TestType Unit -Project Xaml -TestFilter "Maui12345" + # Run a specific XAML unit test + +.EXAMPLE + ./RunTests.ps1 -TestType Unit + # Run ALL unit test projects + +.EXAMPLE + ./RunTests.ps1 -TestType Device -Project Controls -Platform ios + # Run Controls device tests on iOS simulator + +.EXAMPLE + ./RunTests.ps1 -TestType Device -Project Core -Platform android -TestFilter "Category=Button" + # Run Core device tests on Android filtered by category + +.EXAMPLE + ./RunTests.ps1 -TestType UI -Platform android -TestFilter "FullyQualifiedName~Issue12345" + # Run a specific UI test on Android + +.EXAMPLE + ./RunTests.ps1 -TestType UI -Platform ios -Category "SafeAreaEdges" + # Run UI tests by category on iOS + +.EXAMPLE + ./RunTests.ps1 -TestType Integration -Category Build + # Run integration tests for the Build category +#> + +[CmdletBinding(DefaultParameterSetName = "Run")] +param( + [Parameter(Mandatory = $true, ParameterSetName = "Run")] + [ValidateSet("Unit", "Device", "UI", "Integration")] + [string]$TestType, + + [Parameter(ParameterSetName = "Run")] + [string]$Project, + + [Parameter(ParameterSetName = "Run")] + [ValidateSet("android", "ios", "catalyst", "maccatalyst", "windows")] + [string]$Platform, + + [Parameter(ParameterSetName = "Run")] + [string]$TestFilter, + + [Parameter(ParameterSetName = "Run")] + [string]$Category, + + [Parameter(ParameterSetName = "Run")] + [ValidateSet("Debug", "Release")] + [string]$Configuration, + + [Parameter(ParameterSetName = "Run")] + [string]$DeviceUdid, + + [Parameter(ParameterSetName = "Run")] + [switch]$Rebuild, + + [Parameter(ParameterSetName = "Run")] + [string]$ResultsDirectory, + + [Parameter(Mandatory = $true, ParameterSetName = "List")] + [switch]$List +) + +$ErrorActionPreference = "Stop" +$RepoRoot = Resolve-Path "$PSScriptRoot/../.." + +# Import shared utilities +. "$PSScriptRoot/shared/shared-utils.ps1" + +#region Project Definitions + +# Unit test projects: Key → relative csproj path +$UnitTestProjects = [ordered]@{ + "Controls" = "src/Controls/tests/Core.UnitTests/Controls.Core.UnitTests.csproj" + "Xaml" = "src/Controls/tests/Xaml.UnitTests/Controls.Xaml.UnitTests.csproj" + "BindingSourceGen" = "src/Controls/tests/BindingSourceGen.UnitTests/Controls.BindingSourceGen.UnitTests.csproj" + "SourceGen" = "src/Controls/tests/SourceGen.UnitTests/SourceGen.UnitTests.csproj" + "ControlsDesign" = "src/Controls/tests/Core.Design.UnitTests/Controls.Core.Design.UnitTests.csproj" + "Core" = "src/Core/tests/UnitTests/Core.UnitTests.csproj" + "Essentials" = "src/Essentials/test/UnitTests/Essentials.UnitTests.csproj" + "Graphics" = "src/Graphics/tests/Graphics.Tests/Graphics.Tests.csproj" + "Resizetizer" = "src/SingleProject/Resizetizer/test/UnitTests/Resizetizer.UnitTests.csproj" + "Compatibility" = "src/Compatibility/Core/tests/Compatibility.UnitTests/Compatibility.Core.UnitTests.csproj" + "DualScreen" = "src/Controls/Foldable/test/Controls.DualScreen.UnitTests.csproj" + "AI" = "src/AI/tests/Essentials.AI.UnitTests/Essentials.AI.UnitTests.csproj" +} + +# Device test projects: Key → matches Run-DeviceTests.ps1 -Project parameter +$DeviceTestProjects = [ordered]@{ + "Controls" = "src/Controls/tests/DeviceTests/Controls.DeviceTests.csproj" + "Core" = "src/Core/tests/DeviceTests/Core.DeviceTests.csproj" + "Essentials" = "src/Essentials/test/DeviceTests/Essentials.DeviceTests.csproj" + "Graphics" = "src/Graphics/tests/DeviceTests/Graphics.DeviceTests.csproj" + "BlazorWebView"= "src/BlazorWebView/tests/DeviceTests/MauiBlazorWebView.DeviceTests.csproj" + "AI" = "src/AI/tests/Essentials.AI.DeviceTests/Essentials.AI.DeviceTests.csproj" +} + +# Integration test categories +$IntegrationCategories = @( + "Build", "WindowsTemplates", "macOSTemplates", "Blazor", + "MultiProject", "Samples", "AOT", "RunOnAndroid", "RunOniOS" +) + +#endregion + +#region List Mode + +if ($List) { + Write-Host @" + +╔═══════════════════════════════════════════════════════════╗ +║ .NET MAUI Unified Test Runner ║ +║ Available Test Types & Projects ║ +╚═══════════════════════════════════════════════════════════╝ + +"@ -ForegroundColor Magenta + + # Unit Tests + Write-Host " Unit Tests " -ForegroundColor Black -BackgroundColor Cyan + Write-Host " Runs on build machine via dotnet test. No device required." -ForegroundColor Gray + Write-Host "" + Write-Host " Project Key Test Project" -ForegroundColor Yellow + Write-Host " ──────────────── ──────────────────────────────────────────" -ForegroundColor DarkGray + foreach ($key in $UnitTestProjects.Keys) { + Write-Host " $($key.PadRight(20))" -ForegroundColor White -NoNewline + Write-Host "$($UnitTestProjects[$key])" -ForegroundColor Gray + } + Write-Host "" + Write-Host " Usage:" -ForegroundColor DarkCyan + Write-Host " ./RunTests.ps1 -TestType Unit -Project Controls" -ForegroundColor White + Write-Host " ./RunTests.ps1 -TestType Unit -Project Xaml -TestFilter `"Maui12345`"" -ForegroundColor White + Write-Host " ./RunTests.ps1 -TestType Unit # run ALL" -ForegroundColor White + Write-Host "" + + # Device Tests + Write-Host " Device Tests " -ForegroundColor Black -BackgroundColor Green + Write-Host " Runs on device/emulator via xharness. Requires -Platform." -ForegroundColor Gray + Write-Host "" + Write-Host " Project Key Test Project" -ForegroundColor Yellow + Write-Host " ──────────────── ──────────────────────────────────────────" -ForegroundColor DarkGray + foreach ($key in $DeviceTestProjects.Keys) { + Write-Host " $($key.PadRight(20))" -ForegroundColor White -NoNewline + Write-Host "$($DeviceTestProjects[$key])" -ForegroundColor Gray + } + Write-Host "" + Write-Host " Platforms: android, ios, maccatalyst, windows" -ForegroundColor Gray + Write-Host " Usage:" -ForegroundColor DarkCyan + Write-Host " ./RunTests.ps1 -TestType Device -Project Controls -Platform ios" -ForegroundColor White + Write-Host " ./RunTests.ps1 -TestType Device -Project Core -Platform android -TestFilter `"Category=Button`"" -ForegroundColor White + Write-Host "" + + # UI Tests + Write-Host " UI Tests " -ForegroundColor Black -BackgroundColor Yellow + Write-Host " Appium-based UI automation. Requires -Platform and -TestFilter or -Category." -ForegroundColor Gray + Write-Host "" + Write-Host " Builds TestCases.HostApp, deploys to device, runs NUnit tests via Appium." -ForegroundColor Gray + Write-Host " Platforms: android, ios, catalyst, windows" -ForegroundColor Gray + Write-Host "" + Write-Host " Usage:" -ForegroundColor DarkCyan + Write-Host " ./RunTests.ps1 -TestType UI -Platform android -TestFilter `"FullyQualifiedName~Issue12345`"" -ForegroundColor White + Write-Host " ./RunTests.ps1 -TestType UI -Platform ios -Category `"SafeAreaEdges`"" -ForegroundColor White + Write-Host "" + + # Integration Tests + Write-Host " Integration Tests " -ForegroundColor Black -BackgroundColor DarkYellow + Write-Host " Template builds, workload validation. Requires -Category or -TestFilter." -ForegroundColor Gray + Write-Host "" + Write-Host " Categories: $($IntegrationCategories -join ', ')" -ForegroundColor Gray + Write-Host "" + Write-Host " Usage:" -ForegroundColor DarkCyan + Write-Host " ./RunTests.ps1 -TestType Integration -Category Build" -ForegroundColor White + Write-Host " ./RunTests.ps1 -TestType Integration -Category macOSTemplates" -ForegroundColor White + Write-Host "" + + exit 0 +} + +#endregion + +#region Parameter Validation + +# Validate TestFilter and Category are not both specified +if ($TestFilter -and $Category) { + Write-Error "Cannot specify both -TestFilter and -Category. Use one or the other." + exit 1 +} + +# Build the effective filter +$effectiveFilter = $null +if ($Category) { + $effectiveFilter = "Category=$Category" +} elseif ($TestFilter) { + $effectiveFilter = $TestFilter +} + +# Set default Configuration per test type if not specified +if (-not $Configuration) { + $Configuration = switch ($TestType) { + "Device" { "Release" } + default { "Debug" } + } +} + +# Set default results directory +if (-not $ResultsDirectory) { + $ResultsDirectory = Join-Path $RepoRoot "artifacts/test-results" +} + +# Validate platform requirement for Device and UI tests +if ($TestType -eq "Device" -and -not $Platform) { + Write-Error "Device tests require -Platform (android, ios, maccatalyst, windows)" + exit 1 +} +if ($TestType -eq "UI" -and -not $Platform) { + Write-Error "UI tests require -Platform (android, ios, catalyst, windows)" + exit 1 +} + +# Validate filter requirement for UI tests +if ($TestType -eq "UI" -and -not $effectiveFilter) { + Write-Error "UI tests require -TestFilter or -Category" + exit 1 +} + +# Validate filter/category requirement for Integration tests +if ($TestType -eq "Integration" -and -not $effectiveFilter) { + Write-Error "Integration tests require -Category or -TestFilter" + exit 1 +} + +# Validate Project for Device tests +if ($TestType -eq "Device") { + if (-not $Project) { + Write-Error "Device tests require -Project ($($DeviceTestProjects.Keys -join ', '))" + exit 1 + } + if (-not $DeviceTestProjects.Contains($Project)) { + Write-Error "Unknown device test project '$Project'. Valid values: $($DeviceTestProjects.Keys -join ', ')" + exit 1 + } +} + +# Validate Project for Unit tests (if specified) +if ($TestType -eq "Unit" -and $Project -and -not $UnitTestProjects.Contains($Project)) { + Write-Error "Unknown unit test project '$Project'. Valid values: $($UnitTestProjects.Keys -join ', ')" + exit 1 +} + +#endregion + +#region Banner + +$testTypeDisplay = switch ($TestType) { + "Unit" { "UNIT TESTS" } + "Device" { "DEVICE TESTS" } + "UI" { "UI TESTS" } + "Integration" { "INTEGRATION TESTS" } +} + +$projectDisplay = if ($Project) { $Project } elseif ($TestType -eq "Unit") { "ALL" } else { "-" } +$platformDisplay = if ($Platform) { $Platform.ToUpper() } else { "N/A" } +$filterDisplay = if ($effectiveFilter) { $effectiveFilter } else { "None" } + +Write-Host @" + +╔═══════════════════════════════════════════════════════════╗ +║ .NET MAUI Unified Test Runner ║ +╠═══════════════════════════════════════════════════════════╣ +║ Test Type: $($testTypeDisplay.PadRight(42))║ +║ Project: $($projectDisplay.PadRight(42))║ +║ Platform: $($platformDisplay.PadRight(42))║ +║ Filter: $($filterDisplay.Substring(0, [Math]::Min(42, $filterDisplay.Length)).PadRight(42))║ +║ Config: $($Configuration.PadRight(42))║ +╚═══════════════════════════════════════════════════════════╝ + +"@ -ForegroundColor Magenta + +#endregion + +#region Run Unit Tests + +function Invoke-UnitTests { + param( + [string]$ProjectKey, + [string]$Filter, + [string]$Config + ) + + # Determine which projects to run + $projectsToRun = [ordered]@{} + if ($ProjectKey) { + $projectsToRun[$ProjectKey] = $UnitTestProjects[$ProjectKey] + } else { + $projectsToRun = $UnitTestProjects + } + + Write-Step "Running $($projectsToRun.Count) unit test project(s)..." + + # Ensure results directory exists + if (-not (Test-Path $ResultsDirectory)) { + New-Item -Path $ResultsDirectory -ItemType Directory -Force | Out-Null + } + + $totalPassed = 0 + $totalFailed = 0 + $totalSkipped = 0 + $failedProjects = @() + $results = @() + + foreach ($key in $projectsToRun.Keys) { + $projectPath = Join-Path $RepoRoot $projectsToRun[$key] + + if (-not (Test-Path $projectPath)) { + Write-Warn "Project not found: $projectPath — skipping" + continue + } + + Write-Host "" + Write-Host "─────────────────────────────────────────────────────" -ForegroundColor DarkGray + Write-Step "Testing: $key" + Write-Info "Project: $($projectsToRun[$key])" + + $testArgs = @( + "test", $projectPath, + "--configuration", $Config, + "--no-restore", + "--logger", "console;verbosity=normal", + "--logger", "trx;LogFileName=$key.trx", + "--results-directory", $ResultsDirectory + ) + + if ($Filter) { + $testArgs += @("--filter", $Filter) + } + + $startTime = Get-Date + $output = & dotnet @testArgs 2>&1 + $exitCode = $LASTEXITCODE + $duration = (Get-Date) - $startTime + + # Parse test counts from output + # dotnet test outputs lines like: + # "Total tests: 65" + # " Passed: 65" + # " Failed: 2" + # " Skipped: 3" + # Or single-line: "Passed! - Failed: 0, Passed: 123, Skipped: 0, Total: 123" + $passed = 0 + $failed = 0 + $skipped = 0 + + foreach ($line in $output) { + $lineStr = "$line" + # Match standalone summary lines (with leading whitespace) + if ($lineStr -match "^\s+Passed:\s*(\d+)") { $passed = [int]$Matches[1] } + if ($lineStr -match "^\s+Failed:\s*(\d+)") { $failed = [int]$Matches[1] } + if ($lineStr -match "^\s+Skipped:\s*(\d+)") { $skipped = [int]$Matches[1] } + # Match single-line summary: "Failed: 0, Passed: 123, Skipped: 0" + if ($lineStr -match "Failed:\s*(\d+),\s*Passed:\s*(\d+),\s*Skipped:\s*(\d+)") { + $failed = [int]$Matches[1] + $passed = [int]$Matches[2] + $skipped = [int]$Matches[3] + } + } + + $totalPassed += $passed + $totalFailed += $failed + $totalSkipped += $skipped + + $result = @{ + Project = $key + Passed = $passed + Failed = $failed + Skipped = $skipped + Duration = $duration + ExitCode = $exitCode + } + $results += $result + + if ($exitCode -ne 0) { + $failedProjects += $key + Write-Error "$key — FAILED ($failed failed, $passed passed) [$([math]::Round($duration.TotalSeconds))s]" + # Show failure output + $output | Where-Object { $_ -match "Failed|Error|Exception" } | Select-Object -First 20 | ForEach-Object { + Write-Host " $_" -ForegroundColor Red + } + } else { + Write-Success "$key — PASSED ($passed passed, $skipped skipped) [$([math]::Round($duration.TotalSeconds))s]" + } + } + + # Summary + Write-Host "" + Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Cyan + Write-Host " Unit Test Summary " -ForegroundColor Cyan + Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Cyan + Write-Host "" + + # Results table + Write-Host " Project Passed Failed Skipped Time" -ForegroundColor Yellow + Write-Host " ──────────────────── ─────── ─────── ─────── ─────" -ForegroundColor DarkGray + foreach ($r in $results) { + $statusColor = if ($r.ExitCode -eq 0) { "Green" } else { "Red" } + $line = " $($r.Project.PadRight(22)) $($r.Passed.ToString().PadLeft(7)) $($r.Failed.ToString().PadLeft(7)) $($r.Skipped.ToString().PadLeft(7)) $([math]::Round($r.Duration.TotalSeconds))s" + Write-Host $line -ForegroundColor $statusColor + } + + Write-Host " ──────────────────── ─────── ─────── ─────── ─────" -ForegroundColor DarkGray + $totalColor = if ($totalFailed -eq 0) { "Green" } else { "Red" } + Write-Host " $("TOTAL".PadRight(22)) $($totalPassed.ToString().PadLeft(7)) $($totalFailed.ToString().PadLeft(7)) $($totalSkipped.ToString().PadLeft(7))" -ForegroundColor $totalColor + Write-Host "" + + if ($failedProjects.Count -gt 0) { + Write-Host " ❌ FAILED projects: $($failedProjects -join ', ')" -ForegroundColor Red + } else { + Write-Host " ✅ All unit tests passed" -ForegroundColor Green + } + + Write-Host "" + Write-Host " Results: $ResultsDirectory" -ForegroundColor Gray + Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Cyan + + return ($failedProjects.Count -eq 0) +} + +#endregion + +#region Run Device Tests + +function Invoke-DeviceTests { + $deviceTestScript = Join-Path $RepoRoot ".github/skills/run-device-tests/scripts/Run-DeviceTests.ps1" + + if (-not (Test-Path $deviceTestScript)) { + Write-Error "Device test script not found: $deviceTestScript" + return $false + } + + Write-Step "Delegating to Run-DeviceTests.ps1..." + Write-Info "Project: $Project, Platform: $Platform" + + # Normalize platform for the device test script + $devicePlatform = $Platform + if ($devicePlatform -eq "catalyst") { $devicePlatform = "maccatalyst" } + + $params = @{ + Project = $Project + Platform = $devicePlatform + Configuration = $Configuration + } + + if ($effectiveFilter) { + $params.TestFilter = $effectiveFilter + } + if ($DeviceUdid) { + $params.DeviceUdid = $DeviceUdid + } + + & $deviceTestScript @params + return ($LASTEXITCODE -eq 0) +} + +#endregion + +#region Run UI Tests + +function Invoke-UITests { + $uiTestScript = Join-Path $RepoRoot ".github/scripts/BuildAndRunHostApp.ps1" + + if (-not (Test-Path $uiTestScript)) { + Write-Error "UI test script not found: $uiTestScript" + return $false + } + + Write-Step "Delegating to BuildAndRunHostApp.ps1..." + Write-Info "Platform: $Platform, Filter: $effectiveFilter" + + $params = @{ + Platform = $Platform + Configuration = $Configuration + } + + if ($Category) { + $params.Category = $Category + } else { + $params.TestFilter = $TestFilter + } + + if ($DeviceUdid) { + $params.DeviceUdid = $DeviceUdid + } + if ($Rebuild) { + $params.Rebuild = $true + } + + & $uiTestScript @params + return ($LASTEXITCODE -eq 0) +} + +#endregion + +#region Run Integration Tests + +function Invoke-IntegrationTests { + $integrationScript = Join-Path $RepoRoot ".github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1" + + if (-not (Test-Path $integrationScript)) { + Write-Error "Integration test script not found: $integrationScript" + return $false + } + + Write-Step "Delegating to Run-IntegrationTests.ps1..." + + $params = @{ + Configuration = $Configuration + SkipBuild = $true + SkipInstall = $true + } + + if ($Category) { + $params.Category = $Category + } elseif ($TestFilter) { + $params.TestFilter = $TestFilter + } + + if ($ResultsDirectory) { + $params.ResultsDirectory = $ResultsDirectory + } + + & $integrationScript @params + return ($LASTEXITCODE -eq 0) +} + +#endregion + +#region Main Execution + +$success = $false + +switch ($TestType) { + "Unit" { + $success = Invoke-UnitTests -ProjectKey $Project -Filter $effectiveFilter -Config $Configuration + } + "Device" { + $success = Invoke-DeviceTests + } + "UI" { + $success = Invoke-UITests + } + "Integration" { + $success = Invoke-IntegrationTests + } +} + +if (-not $success) { + exit 1 +} + +#endregion diff --git a/.github/scripts/post-ai-summary-comment.ps1 b/.github/scripts/post-ai-summary-comment.ps1 index 5b10707cc743..574cd2873457 100644 --- a/.github/scripts/post-ai-summary-comment.ps1 +++ b/.github/scripts/post-ai-summary-comment.ps1 @@ -1,22 +1,17 @@ #!/usr/bin/env pwsh <# .SYNOPSIS - Posts or updates the agent review comment on a GitHub Pull Request with validation. + Posts or updates the AI review summary comment on a GitHub Pull Request. .DESCRIPTION - Creates ONE comment for the entire PR review with all phases wrapped in an expandable section. + Creates ONE comment for the PR review with phases wrapped in expandable sections. Uses HTML marker for identification. - - Content is always auto-loaded from PRAgent phase files - (CustomAgentLogsTmp/PRState//PRAgent/*/content.md). - - **Validates that phases marked as COMPLETE actually have content.** - - Format: - ## 🤖 AI Summary — ✅ APPROVE -
📊 Expand Full Review - Status table + all 4 phases as nested details -
+ Always overwrites the existing comment (no session history). + + Content is auto-loaded from PRAgent phase files: + CustomAgentLogsTmp/PRState//PRAgent/{pre-flight,try-fix,report}/content.md + + Gate is posted separately by post-gate-comment.ps1. .PARAMETER PRNumber The pull request number (required) @@ -24,9 +19,6 @@ .PARAMETER DryRun Print comment instead of posting -.PARAMETER SkipValidation - Skip validation checks (not recommended) - .EXAMPLE ./post-ai-summary-comment.ps1 -PRNumber 12345 @@ -35,31 +27,21 @@ #> param( - [Parameter(Mandatory=$false)] + [Parameter(Mandatory = $true)] [int]$PRNumber, - [Parameter(Mandatory=$false)] - [switch]$DryRun, - - [Parameter(Mandatory=$false)] - [switch]$SkipValidation, - - [Parameter(Mandatory=$false)] - [string]$PreviewFile + [Parameter(Mandatory = $false)] + [switch]$DryRun ) $ErrorActionPreference = "Stop" +$MARKER = "" # ============================================================================ -# INPUT VALIDATION +# LOAD PHASE CONTENT # ============================================================================ -if ($PRNumber -eq 0) { - throw "PRNumber is required." -} - -# Auto-load from PRAgent phase files -Write-Host "ℹ️ Auto-loading from PRAgent phase files..." -ForegroundColor Cyan +Write-Host "ℹ️ Loading phase content for PR #$PRNumber..." -ForegroundColor Cyan $PRAgentDir = "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent" if (-not (Test-Path $PRAgentDir)) { @@ -70,671 +52,77 @@ if (-not (Test-Path $PRAgentDir)) { } if (-not (Test-Path $PRAgentDir)) { - Write-Host "" - Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red - Write-Host "║ ⛔ No PRAgent directory found ║" -ForegroundColor Red - Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red - Write-Host "" - Write-Host "Expected directory: $PRAgentDir" -ForegroundColor Yellow - Write-Host "Ensure PRAgent phase files exist." -ForegroundColor Yellow - throw "No PRAgent directory found. Ensure PRAgent/*/content.md files exist." -} - -# Load each phase content file -$phaseFiles = @{ - "pre-flight" = Join-Path $PRAgentDir "pre-flight/content.md" - "gate" = Join-Path $PRAgentDir "gate/content.md" - "try-fix" = Join-Path $PRAgentDir "try-fix/content.md" - "report" = Join-Path $PRAgentDir "report/content.md" -} - -$loadedPhases = @() -$phaseContentMap = @{} - -foreach ($phase in $phaseFiles.GetEnumerator()) { - if (Test-Path $phase.Value) { - $phaseContentMap[$phase.Key] = Get-Content $phase.Value -Raw -Encoding UTF8 - $loadedPhases += $phase.Key - Write-Host " ✅ Loaded: $($phase.Key) ($((Get-Item $phase.Value).Length) bytes)" -ForegroundColor Green - } else { - Write-Host " ⏭️ Skipped: $($phase.Key) (no content.md)" -ForegroundColor Gray - } -} - -if ($loadedPhases.Count -eq 0) { - throw "No phase content files found in $PRAgentDir. Ensure at least one phase has a content.md file." -} - -Write-Host " 📦 Loaded $($loadedPhases.Count) phase(s): $($loadedPhases -join ', ')" -ForegroundColor Cyan - -# Build synthetic Content from phase files in the expected
format -$syntheticParts = @() - -# Determine phase statuses based on which files exist and content -$phaseStatusMap = @{} -foreach ($phase in @("pre-flight", "gate", "try-fix", "report")) { - if ($phaseContentMap.ContainsKey($phase)) { - $phaseStatusMap[$phase] = "✅ COMPLETE" - } else { - $phaseStatusMap[$phase] = "⏳ PENDING" - } + throw "No PRAgent directory found at: $PRAgentDir" } -# Build status table -$statusTable = @" -| Phase | Status | -|-------|--------| -| Pre-Flight | $($phaseStatusMap['pre-flight']) | -| Gate | $($phaseStatusMap['gate']) | -| Fix | $($phaseStatusMap['try-fix']) | -| Report | $($phaseStatusMap['report']) | -"@ -$syntheticParts += $statusTable - -# Build phase sections -if ($phaseContentMap.ContainsKey('pre-flight')) { - $syntheticParts += @" -
📋 Pre-Flight — Issue Summary - -$($phaseContentMap['pre-flight']) - -
-"@ -} - -if ($phaseContentMap.ContainsKey('gate')) { - $syntheticParts += @" -
🚦 Gate — Test Verification - -$($phaseContentMap['gate']) - -
-"@ +$phases = [ordered]@{ + "pre-flight" = @{ File = "pre-flight/content.md"; Icon = "🔍"; Title = "Pre-Flight — Context & Validation" } + "try-fix" = @{ File = "try-fix/content.md"; Icon = "🔧"; Title = "Fix — Analysis & Comparison" } + "report" = @{ File = "report/content.md"; Icon = "📋"; Title = "Report — Final Recommendation" } } -if ($phaseContentMap.ContainsKey('try-fix')) { - $syntheticParts += @" -
🔧 Fix — Analysis & Comparison - -$($phaseContentMap['try-fix']) - -
-"@ -} +$phaseSections = @() +$statusRows = @() + +foreach ($key in $phases.Keys) { + $phase = $phases[$key] + $filePath = Join-Path $PRAgentDir $phase.File + + if (Test-Path $filePath) { + $content = Get-Content $filePath -Raw -Encoding UTF8 + if (-not [string]::IsNullOrWhiteSpace($content)) { + Write-Host " ✅ $key ($((Get-Item $filePath).Length) bytes)" -ForegroundColor Green + $statusRows += "| $($phase.Title -replace ' —.*','') | ✅ COMPLETE |" + $phaseSections += @" +
+$($phase.Icon) $($phase.Title) -if ($phaseContentMap.ContainsKey('report')) { - $syntheticParts += @" -
📋 Report — Final Recommendation +--- -$($phaseContentMap['report']) +$content
"@ -} - -$Content = $syntheticParts -join "`n`n---`n`n" -Write-Host " ✅ Built synthetic content ($($Content.Length) chars)" -ForegroundColor Green - -# Final validation -if ([string]::IsNullOrWhiteSpace($Content)) { - Write-Host "" - Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red - Write-Host "║ ⛔ No content loaded from phase files ║" -ForegroundColor Red - Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red - Write-Host "" - Write-Host "Usage:" -ForegroundColor Yellow - Write-Host " ./post-ai-summary-comment.ps1 -PRNumber 12345 # auto-loads from PRAgent/*/content.md" -ForegroundColor Gray - Write-Host "" - throw "No content loaded from PRAgent phase files." -} - -Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Cyan -Write-Host "║ AI Summary Comment (with Validation) ║" -ForegroundColor Cyan -Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Cyan - -# ============================================================================ -# VALIDATION FUNCTIONS -# ============================================================================ - -function Test-PhaseContentComplete { - param( - [string]$PhaseContent, - [string]$PhaseName, - [string]$PhaseStatus, - [switch]$Debug - ) - - # Skip validation if phase is not marked COMPLETE or PASSED - if ($PhaseStatus -notmatch '✅\s*(COMPLETE|PASSED)') { - return @{ IsValid = $true; Errors = @(); Warnings = @() } - } - - $validationErrors = @() - $validationWarnings = @() - - # Check if content exists - if ([string]::IsNullOrWhiteSpace($PhaseContent)) { - $validationErrors += "Phase $PhaseName is marked as '$PhaseStatus' but has NO content" - if ($Debug) { - Write-Host " [DEBUG] Content is null or whitespace for phase: $PhaseName" -ForegroundColor DarkGray - } - return @{ IsValid = $false; Errors = $validationErrors; Warnings = @() } - } - - if ($Debug) { - Write-Host " [DEBUG] $PhaseName content length: $($PhaseContent.Length) chars" -ForegroundColor DarkGray - Write-Host " [DEBUG] First 100 chars: $($PhaseContent.Substring(0, [Math]::Min(100, $PhaseContent.Length)))" -ForegroundColor DarkGray - } - - # Check for PENDING markers - only match [PENDING] placeholder markers. - # ⏳ PENDING is a status indicator (redundant with phase table), not an unfilled placeholder. - $pendingMatches = [regex]::Matches($PhaseContent, '\[PENDING\]') - if ($pendingMatches.Count -gt 0) { - $validationErrors += "Phase $PhaseName is marked as '$PhaseStatus' but contains $($pendingMatches.Count) PENDING markers" - } - - # Phase-specific validation (relaxed for better UX) - switch ($PhaseName) { - "Pre-Flight" { - if ($PhaseContent -notmatch 'Platforms Affected:') { - $validationWarnings += "Pre-Flight missing 'Platforms Affected' section (non-critical)" - } - } - "Gate" { - if ($PhaseContent -notmatch 'Result:') { - $validationWarnings += "Gate phase missing 'Result' field (non-critical)" - } - } - "Fix" { - if ($PhaseContent -notmatch 'Selected Fix:') { - $validationErrors += "Fix phase missing 'Selected Fix' field" - } - if ($PhaseContent -notmatch 'Exhausted:') { - $validationWarnings += "Fix phase missing 'Exhausted' field (non-critical)" - } - } - "Report" { - # Relaxed validation - only check for substantive content - $hasRecommendation = $PhaseContent -match '(Final Recommendation|Verdict|Recommendation:|APPROVE|REQUEST CHANGES)' - $hasAnalysis = $PhaseContent -match '(Summary|Fix Quality|Test Quality|Why|Analysis)' - - if (-not $hasRecommendation) { - $validationWarnings += "Report phase missing clear recommendation (non-critical)" - } - if (-not $hasAnalysis) { - $validationWarnings += "Report phase missing analysis sections (non-critical)" - } - - # Only error if content is extremely short - if ($PhaseContent.Length -lt 200) { - $validationErrors += "Report phase content is too short ($($PhaseContent.Length) chars) - expected comprehensive final report" - } - } - } - - return @{ - IsValid = ($validationErrors.Count -eq 0) - Errors = $validationErrors - Warnings = $validationWarnings - } -} - -# ============================================================================ -# EXTRACTION FUNCTIONS -# ============================================================================ - -# Extract recommendation from content -$recommendation = "IN PROGRESS" -if ($Content -match '##\s+✅\s+Final Recommendation:\s+APPROVE') { - $recommendation = "✅ APPROVE" -} elseif ($Content -match '##\s+⚠️\s+Final Recommendation:\s+REQUEST CHANGES') { - $recommendation = "⚠️ REQUEST CHANGES" -} elseif ($Content -match 'Final Recommendation:\s+APPROVE') { - $recommendation = "✅ APPROVE" -} elseif ($Content -match 'Final Recommendation:\s+REQUEST CHANGES') { - $recommendation = "⚠️ REQUEST CHANGES" -} - -# Extract phase statuses from content -$phaseStatuses = @{ - "Pre-Flight" = "⏳ PENDING" - "Gate" = "⏳ PENDING" - "Fix" = "⏳ PENDING" - "Report" = "⏳ PENDING" -} - -# Parse phase status table - match any status format -if ($Content -match '(?s)\|\s*Phase\s*\|\s*Status\s*\|.*?\n\|[\s-]+\|[\s-]+\|(.*?)(?=\n\n|---|\z)') { - $tableContent = $Matches[1] - $tableContent -split '\n' | ForEach-Object { - if ($_ -match '\|\s*(.+?)\s*\|\s*(.+?)\s*\|') { - $phaseName = $Matches[1].Trim() -replace '^🔍\s*', '' -replace '^🧪\s*', '' -replace '^🚦\s*', '' -replace '^🔧\s*', '' -replace '^📋\s*', '' - $status = $Matches[2].Trim() - if ($phaseStatuses.ContainsKey($phaseName)) { - $phaseStatuses[$phaseName] = $status - } - } - } -} - -# ============================================================================ -# DYNAMIC SECTION EXTRACTION -# ============================================================================ - -# Extract ALL sections from content dynamically -function Extract-AllSections { - param( - [string]$StateContent, - [switch]$Debug - ) - - $sections = @{} - - # Pattern to find all
TITLE...content...
blocks - # Note: [^>]* handles optional attributes like "open" in
- $pattern = '(?s)]*>\s*([^<]+)(.*?)
' - $matches = [regex]::Matches($StateContent, $pattern) - - if ($Debug) { - Write-Host " [DEBUG] Found $($matches.Count) section(s) in content" -ForegroundColor Cyan - } - - foreach ($match in $matches) { - $title = $match.Groups[1].Value.Trim() - $content = $match.Groups[2].Value.Trim() - - $sections[$title] = $content - - if ($Debug) { - Write-Host " [DEBUG] Section: '$title' (${content.Length} chars)" -ForegroundColor DarkGray - } - } - - return $sections -} - -# Extract all sections dynamically -$debugMode = $false # Set to $true for debugging -if ($DebugPreference -eq 'Continue') { $debugMode = $true } - -$allSections = Extract-AllSections -StateContent $Content -Debug:$debugMode - -# Map sections to phase content using flexible matching -function Get-SectionByPattern { - param( - [hashtable]$Sections, - [string[]]$Patterns, - [switch]$Debug - ) - - foreach ($pattern in $Patterns) { - foreach ($key in $Sections.Keys) { - if ($key -match $pattern) { - if ($Debug) { - Write-Host " [DEBUG] Matched '$key' with pattern '$pattern'" -ForegroundColor Green - } - return $Sections[$key] - } - } - } - - if ($Debug) { - Write-Host " [DEBUG] No match for patterns: $($Patterns -join ', ')" -ForegroundColor Yellow - Write-Host " [DEBUG] Available sections: $($Sections.Keys -join ', ')" -ForegroundColor Yellow - } - - return $null -} - -# Map to phase content with flexible patterns (regex) -$preFlightContent = Get-SectionByPattern -Sections $allSections -Patterns @( - '📋.*Issue Summary', - '📋.*Pre-Flight', - '🔍.*Pre-Flight' -) -Debug:$debugMode - -$gateContent = Get-SectionByPattern -Sections $allSections -Patterns @( - '🚦.*Gate', - '📋.*Gate' -) -Debug:$debugMode - -$fixContent = Get-SectionByPattern -Sections $allSections -Patterns @( - '🔧.*Fix', - '📋.*Fix' -) -Debug:$debugMode - -$reportContent = Get-SectionByPattern -Sections $allSections -Patterns @( - '📋.*Report', - 'Phase 4.*Report', - 'Final Report' -) -Debug:$debugMode - -# Fallback: If Report content not found in
blocks, look for -# "## Final Recommendation" section directly in the markdown (agent sometimes -# writes Report as a top-level heading instead of a
block) -if ([string]::IsNullOrWhiteSpace($reportContent)) { - # Look for "## Final Recommendation" heading - capture up to the first --- separator - # or
block to avoid including content from other phases - if ($Content -match '(?s)##\s+[✅⚠️❌\uFE0F]*\s*Final Recommendation[:\s].+?(?=\n---|\n) -function New-ReviewSession { - param([string]$PhaseContent, [string]$CommitTitle, [string]$CommitSha, [string]$CommitUrl) - - if ([string]::IsNullOrWhiteSpace($PhaseContent)) { - return "" - } - - # Return raw content — the commit info is shown on the top-level summary - return $PhaseContent -} - -# Helper function to extract existing review sessions from a phase -function Get-ExistingReviewSessions { - param([string]$PhaseContent) - - if ([string]::IsNullOrWhiteSpace($PhaseContent)) { - return @() - } - - $sessions = @() - # Try old format first (wrapped in
📝 ...) - $pattern = '(?s)
\s*📝.*?.*?
' - $matches = [regex]::Matches($PhaseContent, $pattern) - - if ($matches.Count -gt 0) { - # Old format: extract the inner content from each session wrapper - foreach ($match in $matches) { - $inner = $match.Value - if ($inner -match '(?s)📝.*?\s*---\s*(.*?)\s*
') { - $sessions += ($Matches[1].Trim() -replace '(?m)^---\s*$', '').Trim() - } else { - $sessions += $inner - } + Write-Host " ⏭️ $key (empty)" -ForegroundColor Gray + $statusRows += "| $($phase.Title -replace ' —.*','') | ⏳ PENDING |" } } else { - # New format: content is directly in the phase section (no wrapper) - # Strip leading/trailing --- separators that may remain from old format - $cleaned = ($PhaseContent.Trim() -replace '(?m)^---\s*$', '').Trim() - $sessions += $cleaned - } - - return $sessions -} - -# Helper function to combine existing sessions with new session -function Merge-ReviewSessions { - param( - [string[]]$ExistingSessions, - [string]$NewSession, - [string]$NewCommitSha - ) - - if ([string]::IsNullOrWhiteSpace($NewSession)) { - return "" - } - - # Check if any existing session is for the same commit - $allSessions = @() - $replaced = $false - - foreach ($existingSession in $ExistingSessions) { - # Check if this session contains the new commit SHA - if ($existingSession -match "$NewCommitSha") { - # Replace this session with the new one (only once) - if (-not $replaced) { - $allSessions += $NewSession - $replaced = $true - } - # Skip the old session with same commit SHA - } else { - # Keep the existing session - $allSessions += $existingSession - } - } - - # If we didn't replace any session, add the new one - if (-not $replaced) { - $allSessions += $NewSession + Write-Host " ⏭️ $key (not found)" -ForegroundColor Gray + $statusRows += "| $($phase.Title -replace ' —.*','') | ⏳ PENDING |" } - - return ($allSessions -join "`n`n---`n`n") } -# Fetch existing comment to preserve old review sessions -Write-Host "Checking for existing review comment..." -ForegroundColor Yellow -$existingComment = gh api "repos/dotnet/maui/issues/$PRNumber/comments" --jq '.[] | select(.body | contains("")) | {id: .id, body: .body}' | ConvertFrom-Json - -$existingPreFlightSessions = @() -$existingTestsSessions = @() -$existingGateSessions = @() -$existingFixSessions = @() -$existingReportSessions = @() - -if ($existingComment) { - Write-Host "✓ Found existing review comment (ID: $($existingComment.id)) - extracting review sessions..." -ForegroundColor Green - - # Helper function to extract phase content with fallback patterns - function Extract-PhaseFromComment { - param( - [string]$CommentBody, - [string]$Emoji, - [string]$PhaseName - ) - - # Try patterns in order of specificity (most specific first) - $patterns = @( - # Pattern 1: Phase name anywhere in the header - "(?s).*?$PhaseName.*?(.*?)
" - # Pattern 2: Just emoji (most lenient fallback) - "(?s)$Emoji[^<]*(.*?)
" - ) - - foreach ($pattern in $patterns) { - if ($CommentBody -match $pattern) { - return $Matches[1] - } - } - - return $null - } - - # Extract existing sessions from each phase with fallback - $preFlightMatch = Extract-PhaseFromComment -CommentBody $existingComment.body -Emoji "🔍" -PhaseName "Pre-Flight" - if ($preFlightMatch) { $existingPreFlightSessions = Get-ExistingReviewSessions -PhaseContent $preFlightMatch } - - $gateMatch = Extract-PhaseFromComment -CommentBody $existingComment.body -Emoji "🚦" -PhaseName "Gate" - if ($gateMatch) { $existingGateSessions = Get-ExistingReviewSessions -PhaseContent $gateMatch } - - $fixMatch = Extract-PhaseFromComment -CommentBody $existingComment.body -Emoji "🔧" -PhaseName "Fix" - if ($fixMatch) { $existingFixSessions = Get-ExistingReviewSessions -PhaseContent $fixMatch } - - $reportMatch = Extract-PhaseFromComment -CommentBody $existingComment.body -Emoji "📋" -PhaseName "Report" - if ($reportMatch) { $existingReportSessions = Get-ExistingReviewSessions -PhaseContent $reportMatch } -} else { - Write-Host "✓ No existing comment found - creating new..." -ForegroundColor Yellow +if ($phaseSections.Count -eq 0) { + throw "No phase content found. Ensure at least one content.md exists in $PRAgentDir." } -# Create NEW review sessions from current content -$newPreFlightSession = New-ReviewSession -PhaseContent $preFlightContent -CommitTitle $latestCommitTitle -CommitSha $latestCommitSha -CommitUrl $latestCommitUrl -$newGateSession = New-ReviewSession -PhaseContent $gateContent -CommitTitle $latestCommitTitle -CommitSha $latestCommitSha -CommitUrl $latestCommitUrl -$newFixSession = New-ReviewSession -PhaseContent $fixContent -CommitTitle $latestCommitTitle -CommitSha $latestCommitSha -CommitUrl $latestCommitUrl -$newReportSession = New-ReviewSession -PhaseContent $reportContent -CommitTitle $latestCommitTitle -CommitSha $latestCommitSha -CommitUrl $latestCommitUrl - -# Merge existing sessions with new session (if new content exists) -$allPreFlightSessions = if ($newPreFlightSession) { Merge-ReviewSessions -ExistingSessions $existingPreFlightSessions -NewSession $newPreFlightSession -NewCommitSha $latestCommitSha } else { $existingPreFlightSessions -join "`n`n---`n`n" } -$allGateSessions = if ($newGateSession) { Merge-ReviewSessions -ExistingSessions $existingGateSessions -NewSession $newGateSession -NewCommitSha $latestCommitSha } else { $existingGateSessions -join "`n`n---`n`n" } -$allFixSessions = if ($newFixSession) { Merge-ReviewSessions -ExistingSessions $existingFixSessions -NewSession $newFixSession -NewCommitSha $latestCommitSha } else { $existingFixSessions -join "`n`n---`n`n" } -$allReportSessions = if ($newReportSession) { Merge-ReviewSessions -ExistingSessions $existingReportSessions -NewSession $newReportSession -NewCommitSha $latestCommitSha } else { $existingReportSessions -join "`n`n---`n`n" } - -# Build phase sections dynamically - only include phases with content -$phaseSections = @() - -# Helper to create phase section -function New-PhaseSection { - param( - [string]$Icon, - [string]$PhaseName, - [string]$Subtitle, - [string]$Content, - [string]$Status - ) - - # Skip phases with no content - if ([string]::IsNullOrWhiteSpace($Content) -or $Content -eq "_No review sessions yet_") { - return $null - } - - return @" -
-$Icon $PhaseName — $Subtitle - ---- - -$Content +# ============================================================================ +# BUILD COMMENT BODY +# ============================================================================ -
-"@ +# Get latest commit info for the summary header +try { + $commitJson = gh api "repos/dotnet/maui/pulls/$PRNumber/commits" --jq '.[-1] | {message: .commit.message, sha: .sha}' 2>$null | ConvertFrom-Json +} catch { + Write-Host "⚠️ Failed to fetch commit info: $_" -ForegroundColor Yellow + $commitJson = $null } +$commitTitle = if ($commitJson) { ($commitJson.message -split "`n")[0] } else { "Unknown" } +$commitSha = if ($commitJson) { $commitJson.sha.Substring(0, 7) } else { "unknown" } +$commitUrl = if ($commitJson) { "https://github.com/dotnet/maui/commit/$($commitJson.sha)" } else { "#" } -# Build phase sections (only non-empty ones) -$preFlightSection = New-PhaseSection -Icon "🔍" -PhaseName "Pre-Flight" -Subtitle "Context & Validation" -Content $allPreFlightSessions -Status $phaseStatuses['Pre-Flight'] -$gateSection = New-PhaseSection -Icon "🚦" -PhaseName "Gate" -Subtitle "Test Verification" -Content $allGateSessions -Status $phaseStatuses['Gate'] -$fixSection = New-PhaseSection -Icon "🔧" -PhaseName "Fix" -Subtitle "Analysis & Comparison" -Content $allFixSessions -Status $phaseStatuses['Fix'] -$reportSection = New-PhaseSection -Icon "📋" -PhaseName "Report" -Subtitle "Final Recommendation" -Content $allReportSessions -Status $phaseStatuses['Report'] - -# Collect non-null sections -if ($preFlightSection) { $phaseSections += $preFlightSection } -if ($gateSection) { $phaseSections += $gateSection } -if ($fixSection) { $phaseSections += $fixSection } -if ($reportSection) { $phaseSections += $reportSection } - -# Join sections with separators -$phaseContent = if ($phaseSections.Count -gt 0) { - $phaseSections -join "`n`n---`n`n" -} else { - "_No phases completed yet_" -} +$statusTable = "| Phase | Status |`n|-------|--------|`n$($statusRows -join "`n")" +$phaseContent = $phaseSections -join "`n`n---`n`n" -# ============================================================================ -# UNIFIED COMMENT HANDLING -# Uses single comment with section markers -# ============================================================================ +$commentBody = @" +$MARKER -$MAIN_MARKER = "" -$SECTION_START = "" -$SECTION_END = "" +## 🤖 AI Summary -# Build the PR review section with markers -$prReviewSection = @" -$SECTION_START +
-📊 Expand Full Review — $latestCommitSha · $latestCommitTitle +📊 Expand Full Review — $commitSha · $commitTitle --- @@ -743,131 +131,61 @@ $phaseContent ---
-$SECTION_END + "@ -# Check if there are other sections in the existing comment that we need to preserve -$existingOtherSections = "" -if ($existingComment) { - $body = $existingComment.body - - # Extract all non-PR-REVIEW sections - $sectionTypes = @("TRY-FIX", "WRITE-TESTS", "PR-FINALIZE") - foreach ($sectionType in $sectionTypes) { - $sStart = [regex]::Escape("") - $sEnd = [regex]::Escape("") - if ($body -match "(?s)($sStart.*?$sEnd)") { - $existingOtherSections += "`n`n" + $Matches[1] - } - } -} +Write-Host " ✅ Built comment ($($commentBody.Length) chars)" -ForegroundColor Green -# Build aggregated comment body -$commentBody = @" -$MAIN_MARKER +# ============================================================================ +# DRY RUN +# ============================================================================ -## 🤖 AI Summary +if ($DryRun) { + Write-Host "" + Write-Host "=== COMMENT PREVIEW ===" -ForegroundColor Cyan + Write-Host $commentBody + Write-Host "=== END PREVIEW ===" -ForegroundColor Cyan + exit 0 +} -$prReviewSection -$existingOtherSections -"@ +# ============================================================================ +# POST OR UPDATE COMMENT +# ============================================================================ -# Clean up any double newlines -$commentBody = $commentBody -replace "`n{4,}", "`n`n`n" +Write-Host "Checking for existing review comment..." -ForegroundColor Yellow +# Use --paginate to search ALL comments (not just first 30), pick the LAST matching one +$existingCommentId = gh api "repos/dotnet/maui/issues/$PRNumber/comments" --paginate ` + --jq "[.[] | select(.body | contains(`"$MARKER`"))] | last | .id" 2>$null -if ($DryRun) { - # File-based DryRun: mirrors GitHub comment behavior using a local file - if ([string]::IsNullOrWhiteSpace($PreviewFile)) { - $PreviewFile = "CustomAgentLogsTmp/PRState/$PRNumber/ai-summary-comment-preview.md" - } - - # Ensure directory exists - $previewDir = Split-Path $PreviewFile -Parent - if (-not (Test-Path $previewDir)) { - New-Item -ItemType Directory -Path $previewDir -Force | Out-Null - } - - # Read existing preview file - $existingPreview = "" - if (Test-Path $PreviewFile) { - $existingPreview = Get-Content $PreviewFile -Raw -Encoding UTF8 - Write-Host "ℹ️ Updating existing preview file: $PreviewFile" -ForegroundColor Cyan - } else { - Write-Host "ℹ️ Creating new preview file: $PreviewFile" -ForegroundColor Cyan - } - - # Update or insert the PR-REVIEW section - $PR_REVIEW_MARKER = "" - $PR_REVIEW_END_MARKER = "" - - if ($existingPreview -match [regex]::Escape($PR_REVIEW_MARKER)) { - # Replace existing PR-REVIEW section - $pattern = [regex]::Escape($PR_REVIEW_MARKER) + "[\s\S]*?" + [regex]::Escape($PR_REVIEW_END_MARKER) - $finalComment = $existingPreview -replace $pattern, $prReviewSection - } elseif (-not [string]::IsNullOrWhiteSpace($existingPreview)) { - # Append PR-REVIEW section to existing content - $finalComment = $existingPreview.TrimEnd() + "`n`n" + $prReviewSection - } else { - # New file - use full comment body - $finalComment = $commentBody - } - - # Write to preview file - Set-Content -Path $PreviewFile -Value "$($finalComment.TrimEnd())`n" -Encoding UTF8 -NoNewline - - Write-Host "`n=== COMMENT PREVIEW ===" -ForegroundColor Yellow - Write-Host $finalComment - Write-Host "`n=== END PREVIEW ===" -ForegroundColor Yellow - Write-Host "`n✅ Preview saved to: $PreviewFile" -ForegroundColor Green - Write-Host " Run 'open $PreviewFile' to view in editor" -ForegroundColor Gray - exit 0 +if ($existingCommentId -eq "null" -or [string]::IsNullOrWhiteSpace($existingCommentId)) { + $existingCommentId = $null } -# Post or update comment (reuse $existingComment from earlier check) -if ($existingComment) { - Write-Host "✓ Updating existing review comment (ID: $($existingComment.id))..." -ForegroundColor Green - - # Create temp file for update - $tempFile = [System.IO.Path]::GetTempFileName() +$tempFile = [System.IO.Path]::GetTempFileName() +try { @{ body = $commentBody } | ConvertTo-Json -Depth 10 | Set-Content -Path $tempFile -Encoding UTF8 - $patchResult = $null - $commentId = $existingComment.id - try { - $patchResult = gh api --method PATCH "repos/dotnet/maui/issues/comments/$($existingComment.id)" --input $tempFile 2>&1 - if ($LASTEXITCODE -ne 0) { throw "PATCH failed (exit code $LASTEXITCODE): $patchResult" } - Write-Host "✅ Review comment updated successfully" -ForegroundColor Green - } catch { - Write-Host "⚠️ Could not update comment (no edit permission?) — creating new comment instead: $_" -ForegroundColor Yellow - - $newTempFile = [System.IO.Path]::GetTempFileName() + if ($existingCommentId) { + Write-Host "✓ Found existing comment (ID: $existingCommentId) — updating..." -ForegroundColor Green try { - @{ body = $commentBody } | ConvertTo-Json -Depth 10 | Set-Content -Path $newTempFile -Encoding UTF8 - $newCommentJson = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $newTempFile - $commentId = ($newCommentJson | ConvertFrom-Json).id - Write-Host "✅ Review comment posted as new (ID: $commentId)" -ForegroundColor Green - } finally { - Remove-Item $newTempFile -ErrorAction SilentlyContinue + gh api --method PATCH "repos/dotnet/maui/issues/comments/$existingCommentId" --input $tempFile 2>&1 | Out-Null + if ($LASTEXITCODE -ne 0) { throw "PATCH failed" } + Write-Host "✅ Review comment updated" -ForegroundColor Green + Write-Output "COMMENT_ID=$existingCommentId" + } catch { + Write-Host "⚠️ Could not update comment $existingCommentId : $_" -ForegroundColor Yellow + $newJson = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile + $newId = ($newJson | ConvertFrom-Json).id + Write-Host "✅ Review comment posted (ID: $newId)" -ForegroundColor Green + Write-Output "COMMENT_ID=$newId" } - } finally { - Remove-Item $tempFile -ErrorAction SilentlyContinue - } - - # Output the comment ID so callers can pass it to subsequent scripts - Write-Output "COMMENT_ID=$commentId" -} else { - Write-Host "Creating new review comment..." -ForegroundColor Yellow - - # Create temp file for new comment - $tempFile = [System.IO.Path]::GetTempFileName() - @{ body = $commentBody } | ConvertTo-Json -Depth 10 | Set-Content -Path $tempFile -Encoding UTF8 - - $newCommentJson = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile - Remove-Item $tempFile - - # Extract and output the new comment ID so callers can pass it to subsequent scripts - $newCommentId = ($newCommentJson | ConvertFrom-Json).id - Write-Host "✅ Review comment posted successfully (ID: $newCommentId)" -ForegroundColor Green - - Write-Output "COMMENT_ID=$newCommentId" + } else { + Write-Host "Creating new review comment..." -ForegroundColor Yellow + $newJson = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile + $newId = ($newJson | ConvertFrom-Json).id + Write-Host "✅ Review comment posted (ID: $newId)" -ForegroundColor Green + Write-Output "COMMENT_ID=$newId" + } +} finally { + Remove-Item $tempFile -ErrorAction SilentlyContinue } diff --git a/.github/scripts/post-gate-comment.ps1 b/.github/scripts/post-gate-comment.ps1 new file mode 100644 index 000000000000..4ed3b2b9bc7f --- /dev/null +++ b/.github/scripts/post-gate-comment.ps1 @@ -0,0 +1,162 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Posts or updates the gate verification comment on a GitHub Pull Request. + +.DESCRIPTION + Creates a separate comment for the gate result, identified by marker. + Reads content from CustomAgentLogsTmp/PRState//PRAgent/gate/content.md. + Updates existing gate comment if found, creates new one otherwise. + +.PARAMETER PRNumber + The pull request number (required) + +.PARAMETER DryRun + Print comment instead of posting + +.EXAMPLE + ./post-gate-comment.ps1 -PRNumber 12345 + +.EXAMPLE + ./post-gate-comment.ps1 -PRNumber 12345 -DryRun +#> + +param( + [Parameter(Mandatory = $true)] + [int]$PRNumber, + + [Parameter(Mandatory = $false)] + [switch]$DryRun +) + +$ErrorActionPreference = "Stop" +$MARKER = "" + +# ============================================================================ +# LOAD GATE CONTENT +# ============================================================================ + +$gateContentPath = "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/gate/content.md" +if (-not (Test-Path $gateContentPath)) { + $repoRoot = git rev-parse --show-toplevel 2>$null + if ($repoRoot) { + $gateContentPath = Join-Path $repoRoot "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/gate/content.md" + } +} + +if (-not (Test-Path $gateContentPath)) { + Write-Host "⚠️ No gate content found at: $gateContentPath" -ForegroundColor Yellow + exit 0 +} + +$gateContent = Get-Content $gateContentPath -Raw -Encoding UTF8 +if ([string]::IsNullOrWhiteSpace($gateContent)) { + Write-Host "⚠️ Gate content is empty" -ForegroundColor Yellow + exit 0 +} + +Write-Host "✅ Loaded gate content ($($gateContent.Length) chars)" -ForegroundColor Green + +# ============================================================================ +# BUILD COMMENT BODY +# ============================================================================ + +# Get latest commit info +try { + $commitJson = gh api "repos/dotnet/maui/pulls/$PRNumber/commits" --jq '.[-1] | {message: .commit.message, sha: .sha}' 2>$null | ConvertFrom-Json +} catch { + Write-Host "⚠️ Failed to fetch commit info: $_" -ForegroundColor Yellow + $commitJson = $null +} +$commitTitle = if ($commitJson) { ($commitJson.message -split "`n")[0] } else { "Unknown" } +$commitSha = if ($commitJson) { $commitJson.sha.Substring(0, 7) } else { "unknown" } +$commitUrl = if ($commitJson) { "https://github.com/dotnet/maui/commit/$($commitJson.sha)" } else { "#" } + +$commentBody = @" +$MARKER + +## 🚦 Gate - Test Before and After Fix + +
+📊 Expand Full Gate — $commitSha · $commitTitle + +--- + +$gateContent + +--- + +
+"@ + +# ============================================================================ +# DRY RUN +# ============================================================================ + +if ($DryRun) { + Write-Host "" + Write-Host "=== GATE COMMENT PREVIEW ===" -ForegroundColor Cyan + Write-Host $commentBody + Write-Host "=== END PREVIEW ===" -ForegroundColor Cyan + exit 0 +} + +# ============================================================================ +# POST OR UPDATE COMMENT +# ============================================================================ + +Write-Host "Checking for existing gate comment..." -ForegroundColor Yellow +# Use --paginate to search ALL comments (not just first 30), pick the LAST matching one +# so we always update the most recent gate comment +$existingCommentId = gh api "repos/dotnet/maui/issues/$PRNumber/comments" --paginate ` + --jq "[.[] | select(.body | contains(`"$MARKER`"))] | last | .id" 2>$null + +# gh --jq returns "null" as a string when the array is empty +if ($existingCommentId -eq "null" -or [string]::IsNullOrWhiteSpace($existingCommentId)) { + $existingCommentId = $null +} + +$tempFile = [System.IO.Path]::GetTempFileName() +try { + @{ body = $commentBody } | ConvertTo-Json -Depth 10 | Set-Content -Path $tempFile -Encoding UTF8 + + if ($existingCommentId) { + Write-Host "✓ Found existing gate comment (ID: $existingCommentId) — updating..." -ForegroundColor Green + try { + gh api --method PATCH "repos/dotnet/maui/issues/comments/$existingCommentId" --input $tempFile 2>&1 | Out-Null + if ($LASTEXITCODE -ne 0) { throw "PATCH failed" } + Write-Host "✅ Gate comment updated" -ForegroundColor Green + Write-Output "COMMENT_ID=$existingCommentId" + } catch { + Write-Host "⚠️ Could not update comment $existingCommentId (may be owned by different user): $_" -ForegroundColor Yellow + # Try to find a comment we CAN update (owned by current authenticated user) + $botLogin = gh api user --jq .login 2>$null + if ($botLogin) { + $ownCommentId = gh api "repos/dotnet/maui/issues/$PRNumber/comments" --paginate ` + --jq "[.[] | select((.body | contains(`"$MARKER`")) and .user.login == `"$botLogin`")] | last | .id" 2>$null + if ($ownCommentId -and $ownCommentId -ne "null") { + Write-Host " Retrying with own comment (ID: $ownCommentId)..." -ForegroundColor Yellow + gh api --method PATCH "repos/dotnet/maui/issues/comments/$ownCommentId" --input $tempFile 2>&1 | Out-Null + if ($LASTEXITCODE -eq 0) { + Write-Host "✅ Gate comment updated (own comment)" -ForegroundColor Green + Write-Output "COMMENT_ID=$ownCommentId" + return + } + } + } + Write-Host " Creating new comment as fallback..." -ForegroundColor Yellow + $newJson = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile + $newId = ($newJson | ConvertFrom-Json).id + Write-Host "✅ Gate comment posted (ID: $newId)" -ForegroundColor Green + Write-Output "COMMENT_ID=$newId" + } + } else { + Write-Host "Creating new gate comment..." -ForegroundColor Yellow + $newJson = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile + $newId = ($newJson | ConvertFrom-Json).id + Write-Host "✅ Gate comment posted (ID: $newId)" -ForegroundColor Green + Write-Output "COMMENT_ID=$newId" + } +} finally { + Remove-Item $tempFile -ErrorAction SilentlyContinue +} diff --git a/.github/scripts/shared/Detect-TestsInDiff.ps1 b/.github/scripts/shared/Detect-TestsInDiff.ps1 new file mode 100644 index 000000000000..d7fdf1c82545 --- /dev/null +++ b/.github/scripts/shared/Detect-TestsInDiff.ps1 @@ -0,0 +1,442 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Detects all tests added/modified in a PR or git diff and classifies them by type. + +.DESCRIPTION + Analyzes changed files to identify each individual test and its type (UITest, UnitTest, + XamlUnitTest, DeviceTest). Returns structured data for each test including the filter + needed to run it and which runner to use. + + Can take a PR number (fetches file list from GitHub) or use the local git diff. + +.PARAMETER PRNumber + GitHub PR number to analyze. Uses `gh` CLI to fetch file list. + +.PARAMETER BaseBranch + Base branch for git diff comparison. If omitted, auto-detected from PR or uses HEAD~1. + +.PARAMETER ChangedFiles + Explicit list of changed file paths (skips PR/git detection). + +.OUTPUTS + Array of hashtables, each with: + - Type: UITest | UnitTest | XamlUnitTest | DeviceTest + - TestName: Human-readable test name (class name or method name) + - Filter: dotnet test --filter value + - Project: Project key for device tests (Controls, Core, etc.) + - ProjectPath: Relative .csproj path for unit tests + - Runner: Which script runs it (BuildAndRunHostApp, dotnet-test, Run-DeviceTests) + - Platform: Whether -Platform is required + - Files: List of test files + +.EXAMPLE + # Detect tests from a PR + ./Detect-TestsInDiff.ps1 -PRNumber 25129 + +.EXAMPLE + # Detect tests from local git diff + ./Detect-TestsInDiff.ps1 -BaseBranch main + +.EXAMPLE + # Pipe explicit file list + ./Detect-TestsInDiff.ps1 -ChangedFiles @("src/Controls/tests/DeviceTests/Editor/EditorTests.iOS.cs") +#> + +param( + [Parameter(Mandatory = $false)] + [string]$PRNumber, + + [Parameter(Mandatory = $false)] + [string]$BaseBranch, + + [Parameter(Mandatory = $false)] + [string[]]$ChangedFiles +) + +$ErrorActionPreference = "Stop" + +# ============================================================ +# Test type classification patterns (ordered by specificity) +# ============================================================ + +$TestTypeRules = @( + @{ + Type = "UITest" + PathPattern = "TestCases\.(Shared\.Tests|HostApp)" + Runner = "BuildAndRunHostApp" + NeedsPlatform = $true + # UI test files come in pairs (HostApp + Shared.Tests). Group by class name. + } + @{ + Type = "XamlUnitTest" + PathPattern = "Xaml\.UnitTests/" + Runner = "dotnet-test" + NeedsPlatform = $false + ProjectPath = "src/Controls/tests/Xaml.UnitTests/Controls.Xaml.UnitTests.csproj" + } + @{ + Type = "DeviceTest" + PathPattern = "DeviceTests/" + Runner = "Run-DeviceTests" + NeedsPlatform = $true + } + @{ + Type = "UnitTest" + PathPattern = "(?30 changed files + $prFiles = gh api "repos/dotnet/maui/pulls/$PRNumber/files" --paginate --jq '.[].filename' 2>$null + if ($LASTEXITCODE -ne 0 -or -not $prFiles) { + $prFiles = gh pr view $PRNumber --json files --jq '.files[].path' 2>$null + } + if ($LASTEXITCODE -ne 0 -or -not $prFiles) { + $prFiles = gh pr diff $PRNumber --name-only 2>$null + } + $ChangedFiles = $prFiles -split "`n" | Where-Object { $_ } + } else { + # Use git diff + $mergeBase = $null + if ($BaseBranch) { + $mergeBase = git merge-base HEAD "origin/$BaseBranch" 2>$null + if (-not $mergeBase) { + $mergeBase = git merge-base HEAD -- "$BaseBranch" 2>$null + } + } + if (-not $mergeBase) { + # Try to detect from PR metadata + try { + $prInfo = gh pr view --json baseRefName --jq '.baseRefName' 2>$null + if ($prInfo) { + $mergeBase = git merge-base HEAD "origin/$prInfo" 2>$null + } + } catch {} + } + if (-not $mergeBase) { + $mergeBase = "HEAD~1" + } + $ChangedFiles = git diff $mergeBase HEAD --name-only 2>$null + } +} + +if (-not $ChangedFiles -or $ChangedFiles.Count -eq 0) { + Write-Host "No changed files detected." -ForegroundColor Yellow + return @() +} + +# ============================================================ +# Step 2: Classify each file and group into test entries +# ============================================================ + +# Infrastructure files to ignore even when in test directories +$IgnoredFileNames = @( + "MauiProgram", "Startup", "TestOptions", "TestCategory", + "AssemblyInfo", "GlobalUsings", "Usings" +) + +# Intermediate: collect test files grouped by type + test name +$testGroups = @{} # Key: "Type:TestName" → Value: hashtable + +foreach ($file in $ChangedFiles) { + # Skip non-code files + if ($file -notmatch "\.(cs|xaml)$") { continue } + # Skip snapshot files + if ($file -match "snapshots/") { continue } + # Skip infrastructure files (MauiProgram.cs, Startup.cs, etc.) + $baseName = [System.IO.Path]::GetFileNameWithoutExtension($file) -replace '\.(iOS|Android|Windows|MacCatalyst)$', '' + if ($baseName -in $IgnoredFileNames) { continue } + + foreach ($rule in $TestTypeRules) { + if ($file -match $rule.PathPattern) { + $testType = $rule.Type + $testName = $null + $project = $null + $projectPath = $null + $filter = $null + + switch ($testType) { + "UITest" { + # Only Shared.Tests files define actual test classes. + # HostApp files are UI pages associated with tests but aren't tests themselves. + if ($file -match "TestCases\.Shared\.Tests") { + if ($file -match "[/\\]([^/\\]+)\.cs$") { + $testName = $matches[1] + } + $filter = $testName + } elseif ($file -match "TestCases\.HostApp") { + # HostApp pages: extract name and associate with Shared.Tests entry + if ($file -match "[/\\]([^/\\]+)\.(cs|xaml)$") { + $testName = $matches[1] + $testName = $testName -replace '\.xaml$', '' + } + # Mark as companion file — will be merged with Shared.Tests entry if one exists + $filter = $testName + } + } + + "XamlUnitTest" { + if ($file -match "[/\\]([^/\\]+)\.(cs|xaml)$") { + $testName = $matches[1] + $testName = $testName -replace '\.(rt|rtsg|rtxc|xaml)$', '' + } + $projectPath = $rule.ProjectPath + $filter = $testName + } + + "DeviceTest" { + if ($file -match "[/\\]([^/\\]+)\.cs$") { + $className = $matches[1] + # Strip platform suffix: EditorTests.iOS → EditorTests + $className = $className -replace '\.(iOS|Android|Windows|MacCatalyst)$', '' + $testName = $className + } + + # Detect which device test project + foreach ($projKey in $DeviceTestProjects.Keys) { + if ($file -like "*$($DeviceTestProjects[$projKey])*") { + $project = $projKey + break + } + } + if (-not $project) { $project = "Controls" } + + # Filter will be set to Method=X after method extraction in Step 4 + $filter = $testName + } + + "UnitTest" { + if ($file -match "[/\\]([^/\\]+)\.cs$") { + $testName = $matches[1] + } + + # Detect which unit test project + foreach ($projName in $UnitTestProjects.Keys) { + if ($file -like "*$($UnitTestProjects[$projName])*") { + $project = $projName + $projectPath = $UnitTestProjectPaths[$projName] + break + } + } + $filter = $testName + } + } + + if ($testName) { + $groupKey = "${testType}:${testName}" + if (-not $testGroups.ContainsKey($groupKey)) { + $testGroups[$groupKey] = @{ + Type = $testType + TestName = $testName + Filter = $filter + Project = $project + ProjectPath = $projectPath + Runner = $rule.Runner + NeedsPlatform = $rule.NeedsPlatform + Files = @() + } + } + $testGroups[$groupKey].Files += $file + } + + break # File matched a rule, don't check further rules + } + } +} + +# ============================================================ +# Step 3: Post-process — remove HostApp-only UI test entries (no test class) +# ============================================================ + +# For UITest entries, verify at least one file is from TestCases.Shared.Tests +foreach ($key in @($testGroups.Keys)) { + $group = $testGroups[$key] + if ($group.Type -ne "UITest") { continue } + + $hasTestClass = $group.Files | Where-Object { $_ -match "TestCases\.Shared\.Tests" } + if (-not $hasTestClass) { + $testGroups.Remove($key) + } +} + +# ============================================================ +# Step 4: For device tests, extract specific test method names from the diff +# for display purposes, but keep the category-based filter +# ============================================================ + +foreach ($key in @($testGroups.Keys)) { + $group = $testGroups[$key] + if ($group.Type -ne "DeviceTest") { continue } + + # Try to find added [Fact] or [Test] methods from the diff + $addedMethods = @() + # Cache PR files API response once before the inner loop + if ($PRNumber -and -not $script:_cachedPRFiles) { + $script:_cachedPRFiles = gh api "repos/dotnet/maui/pulls/$PRNumber/files" --paginate 2>$null | ConvertFrom-Json + if (-not $script:_cachedPRFiles) { $script:_cachedPRFiles = @() } + } + $effectiveMergeBase = if ($mergeBase) { $mergeBase } else { "HEAD~1" } + foreach ($file in $group.Files) { + $patch = $null + if ($PRNumber -and $script:_cachedPRFiles) { + # Look up patch from cached API response + $fileEntry = $script:_cachedPRFiles | Where-Object { $_.filename -eq $file } | Select-Object -First 1 + $patch = if ($fileEntry) { $fileEntry.patch } else { $null } + } elseif (-not $PRNumber) { + # Try from git diff + $patch = git diff $effectiveMergeBase HEAD -- $file 2>$null + } + + if ($patch) { + $addedLines = $patch -split "`n" | Where-Object { $_ -match "^\+" } + foreach ($line in $addedLines) { + if ($line -match "public\s+async\s+Task\s+(\w+)\s*\(" -or + $line -match "public\s+void\s+(\w+)\s*\(") { + $methodName = $matches[1] + if ($methodName -ne "Dispose" -and $methodName -ne "Setup" -and + $addedMethods -notcontains $methodName) { + $addedMethods += $methodName + } + } + } + } + } + + # Extract method names for display, use Category= filter for the device test runner + if ($addedMethods.Count -gt 0) { + $group.TestName = "$($group.TestName) ($($addedMethods -join ', '))" + $group.Methods = $addedMethods + + # Find [Category] attribute from the main (non-platform) test class file + $baseClassName = ($group.TestName -split ' \(')[0] + $repoRoot = git rev-parse --show-toplevel 2>$null + $categoryFilter = $null + + foreach ($file in $group.Files) { + if ($file -match "\.cs$") { + # Try the main class file (without platform suffix) + $testDir = [System.IO.Path]::GetDirectoryName($file) + $mainFile = if ($repoRoot) { Join-Path $repoRoot "$testDir/$baseClassName.cs" } else { $null } + if ($mainFile -and (Test-Path $mainFile)) { + $content = Get-Content $mainFile -Raw -ErrorAction SilentlyContinue + # Match [Category(TestCategory.X)] or [Category("X")] + if ($content -match '\[Category\(TestCategory\.(\w+)\)\]') { + $categoryFilter = "Category=$($matches[1])" + break + } elseif ($content -match '\[Category\("([^"]+)"\)\]') { + $categoryFilter = "Category=$($matches[1])" + break + } + } + # Also check the changed file itself + $fullPath = if ($repoRoot) { Join-Path $repoRoot $file } else { $file } + if (Test-Path $fullPath) { + $content = Get-Content $fullPath -Raw -ErrorAction SilentlyContinue + if ($content -match '\[Category\(TestCategory\.(\w+)\)\]') { + $categoryFilter = "Category=$($matches[1])" + break + } elseif ($content -match '\[Category\("([^"]+)"\)\]') { + $categoryFilter = "Category=$($matches[1])" + break + } + } + } + } + + # Use Category filter if found, otherwise fall back to class name + $group.Filter = if ($categoryFilter) { $categoryFilter } else { $baseClassName } + } +} + +# ============================================================ +# Step 5: Output results +# ============================================================ + +$results = @($testGroups.Values | Sort-Object { $_.Type }, { $_.TestName }) + +if ($results.Count -eq 0) { + Write-Host "No tests detected in changed files." -ForegroundColor Yellow + return @() +} + +# Display summary +Write-Host "" +Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Cyan +Write-Host "║ Detected Tests in PR ║" -ForegroundColor Cyan +Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Cyan +Write-Host "║ Found $($results.Count) test(s) across $($results | Select-Object -ExpandProperty Type -Unique | Measure-Object | Select-Object -ExpandProperty Count) type(s) ║" -ForegroundColor Cyan +Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Cyan +Write-Host "" + +$i = 0 +foreach ($test in $results) { + $i++ + $platformNote = if ($test.NeedsPlatform) { "(requires -Platform)" } else { "" } + $icon = switch ($test.Type) { + "UITest" { "🖥️" } + "DeviceTest" { "📱" } + "UnitTest" { "🧪" } + "XamlUnitTest" { "📄" } + } + + Write-Host " $icon $i. [$($test.Type)] $($test.TestName) $platformNote" -ForegroundColor White + Write-Host " Filter: $($test.Filter)" -ForegroundColor Gray + if ($test.Project) { + Write-Host " Project: $($test.Project)" -ForegroundColor Gray + } + if ($test.ProjectPath) { + Write-Host " Path: $($test.ProjectPath)" -ForegroundColor Gray + } + Write-Host " Runner: $($test.Runner)" -ForegroundColor Gray + Write-Host " Files: $($test.Files -join ', ')" -ForegroundColor DarkGray + Write-Host "" +} + +return $results diff --git a/.github/skills/pr-review/SKILL.md b/.github/skills/pr-review/SKILL.md index a3772b0dbd8c..fa17c664d410 100644 --- a/.github/skills/pr-review/SKILL.md +++ b/.github/skills/pr-review/SKILL.md @@ -1,11 +1,11 @@ --- name: pr-review -description: "End-to-end PR reviewer for dotnet/maui. Orchestrates 4 phases — Pre-Flight, Gate, Try-Fix, Report. Use when asked to 'review PR #XXXXX', 'work on PR #XXXXX', or 'fix issue #XXXXX'." +description: "End-to-end PR reviewer for dotnet/maui. Orchestrates 3 phases — Pre-Flight, Try-Fix, Report. Gate runs separately before this skill. Use when asked to 'review PR #XXXXX', 'work on PR #XXXXX', or 'fix issue #XXXXX'." --- -# PR Review — 4-Phase Orchestrator +# PR Review — 3-Phase Orchestrator -End-to-end PR review workflow that orchestrates phases to verify tests, explore independent fix alternatives, and produce a recommendation. +End-to-end PR review workflow that orchestrates phases to explore independent fix alternatives and produce a recommendation. **Trigger phrases:** "review PR #XXXXX", "work on PR #XXXXX", "fix issue #XXXXX" @@ -17,13 +17,13 @@ End-to-end PR review workflow that orchestrates phases to verify tests, explore ## Overview ``` +Gate (pre-run) → Already completed by Review-PR.ps1 before this skill runs Phase 1: Pre-Flight → Gather context, classify files → .github/pr-review/pr-preflight.md -Phase 2: Gate → ⛔ MUST PASS — verify tests FAIL/PASS → .github/pr-review/pr-gate.md -Phase 3: Try-Fix → ⚠️ MANDATORY multi-model exploration → invoke try-fix skill (×4 models) -Phase 4: Report → Write review recommendation → .github/pr-review/pr-report.md +Phase 2: Try-Fix → ⚠️ MANDATORY multi-model exploration → invoke try-fix skill (×4 models) +Phase 3: Report → Write review recommendation → .github/pr-review/pr-report.md ``` -> **Branch setup** is handled by `Review-PR.ps1` before this skill is invoked. By the time this skill runs, the review branch already exists with the PR commits cherry-picked and squashed. +> **Gate and Branch setup** are handled by `Review-PR.ps1` before this skill is invoked. The gate result is passed in the prompt. Do NOT re-run gate verification. **All phases write output to:** `CustomAgentLogsTmp/PRState/{PRNumber}/PRAgent/{phase}/content.md` @@ -34,11 +34,13 @@ Phase 4: Report → Write review recommendation → .g - ❌ Never run `git checkout` or `git switch` to change branches — stay on the review branch set up by the caller - ❌ Never stop and ask the user — use best judgment to skip blocked phases and continue - ❌ Never mark a phase complete with pending fields -- ❌ **Never skip Phase 3 multi-model exploration — it is MANDATORY for every review, no exceptions** +- ❌ **Never skip Phase 2 multi-model exploration — it is MANDATORY for every review, no exceptions** - ❌ Never run git commands that change branch state during Phases 2-3 (scripts handle file manipulation) +- ❌ **Never duplicate phase content** — each phase writes ONLY to its own `content.md`. Do NOT copy gate results into try-fix or report content files. - ✅ Always create `CustomAgentLogsTmp/` output files for every phase - ✅ Always include `Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>` in any commits - ✅ Always use skills' scripts — don't bypass with manual commands +- ✅ Each phase's `content.md` must use the **exact template** from the phase instruction doc — no extra prose ### Multi-Model Configuration @@ -75,21 +77,7 @@ Gather context from the issue, PR, comments, and classify changed files. --- -## Phase 2: Gate - -> Read and follow `.github/pr-review/pr-gate.md` - -Verify that the PR's tests actually catch the bug (FAIL without fix, PASS with fix). - -**Gate:** Pre-Flight must be ✅ COMPLETE. - -**If Gate fails:** -- Tests PASS without fix → Tests don't catch the bug. Proceed to Try-Fix anyway. -- Tests FAIL with fix → PR's fix doesn't work. Skip Try-Fix, proceed to Report. - ---- - -## Phase 3: Try-Fix → Invoke `try-fix` Skill (×4 Models) +## Phase 2: Try-Fix → Invoke `try-fix` Skill (×4 Models) > Read and follow `.github/skills/try-fix/SKILL.md` @@ -127,7 +115,7 @@ prompt: | Invoke the try-fix skill for PR #XXXXX: - problem: {bug description from Pre-Flight} - platform: {platform from Platform Selection} - - test_command: pwsh .github/scripts/BuildAndRunHostApp.ps1 -Platform {platform} -TestFilter "IssueXXXXX" + - test_command: {test command from detected test type — use BuildAndRunHostApp.ps1 for UITest, Run-DeviceTests.ps1 for DeviceTest, dotnet test for UnitTest} - target_files: - src/{area}/{file1}.cs - src/{area}/{file2}.cs @@ -202,7 +190,7 @@ Write `content.md`: --- -## Phase 4: Report +## Phase 3: Report > Read and follow `.github/pr-review/pr-report.md` diff --git a/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 b/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 index 3f5620b79684..db77a90210be 100644 --- a/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 +++ b/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 @@ -178,7 +178,7 @@ $PlatformConfigs = @{ } "windows" = @{ Tfm = "net10.0-windows10.0.19041.0" - RuntimeIdentifier = "win10-x64" + RuntimeIdentifier = "win-x64" AppExtension = ".exe" XHarnessTarget = $null UsesXHarness = $false @@ -298,6 +298,7 @@ try { "windows" { $buildArgs += "/p:WindowsPackageType=None" $buildArgs += "/p:WindowsAppSDKSelfContained=true" + $buildArgs += "/p:UseMonoRuntime=false" } } @@ -350,7 +351,13 @@ try { } } "windows" { - $appPath = "artifacts/bin/$artifactName/$Configuration/$tfmFolder/$ridFolder/$appName.exe" + $exeSearchPath = "artifacts/bin/$artifactName/$Configuration/$tfmFolder" + $exeFile = Get-ChildItem -Path $exeSearchPath -Filter "$appName.exe" -Recurse -ErrorAction SilentlyContinue | Select-Object -First 1 + if ($exeFile) { + $appPath = $exeFile.FullName + } else { + $appPath = "$exeSearchPath/$ridFolder/$appName.exe" + } } } @@ -590,9 +597,10 @@ try { $passCount = ([regex]::Matches($logContent, '\[PASS\]')).Count $failCount = ([regex]::Matches($logContent, '\[FAIL\]')).Count + # Use Write-Output for results so they're captured by callers (not just Write-Host) Write-Host "" - Write-Host " Passed: $passCount" -ForegroundColor Green - Write-Host " Failed: $failCount" -ForegroundColor $(if ($failCount -gt 0) { "Red" } else { "Green" }) + Write-Output " Passed: $passCount" + Write-Output " Failed: $failCount" Write-Host "" Write-Host " Log file: $($logFile.FullName)" -ForegroundColor Gray @@ -609,11 +617,11 @@ try { Write-Host "" if ($testExitCode -eq 0) { Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Green - Write-Host " Tests completed successfully" -ForegroundColor Green + Write-Output " Tests completed successfully" Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Green } else { Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Yellow - Write-Host " Tests completed with exit code: $testExitCode" -ForegroundColor Yellow + Write-Output " Tests completed with exit code: $testExitCode" Write-Host "═══════════════════════════════════════════════════════════" -ForegroundColor Yellow } diff --git a/.github/skills/try-fix/SKILL.md b/.github/skills/try-fix/SKILL.md index 15dbd5ff219a..ca1fcdfc4089 100644 --- a/.github/skills/try-fix/SKILL.md +++ b/.github/skills/try-fix/SKILL.md @@ -52,7 +52,7 @@ All inputs are provided by the invoker (CI, agent, or user). | Input | Required | Description | |-------|----------|-------------| | Problem | Yes | Description of the bug/issue to fix | -| Test command | Yes | **Repository-specific script** to build, deploy, and test (e.g., `pwsh .github/scripts/BuildAndRunHostApp.ps1 -Platform android -TestFilter "Issue12345"`). **ALWAYS use this script - NEVER manually build/compile.** | +| Test command | Yes | **Repository-specific script** to build and test. Use `BuildAndRunHostApp.ps1` for UI tests, `Run-DeviceTests.ps1` for device tests, or `dotnet test` for unit tests. The correct command is determined by the test type detected in the PR. **ALWAYS use the appropriate script - NEVER manually build/compile.** | | Target files | Yes | Files to investigate for the fix | | Platform | Yes | Target platform (`android`, `ios`, `windows`, `maccatalyst`) | | Hints | Optional | Suggested approaches, prior attempts, or areas to focus on | @@ -265,15 +265,21 @@ Implement your fix. Use `git status --short` and `git diff` to track changes. 🚨 **CRITICAL: ALWAYS use the provided test command script - NEVER manually build/compile.** -**For .NET MAUI repository:** Use `BuildAndRunHostApp.ps1` which handles: -- Building the project -- Deploying to device/simulator -- Running tests -- Capturing logs +**For .NET MAUI repository:** Use the test script matching the test type: + +| Test Type | Command | +|-----------|---------| +| UITest | `pwsh .github/scripts/BuildAndRunHostApp.ps1 -Platform -TestFilter ""` | +| DeviceTest | `pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project -Platform -TestFilter ""` | +| UnitTest | `dotnet test --filter ""` | ```powershell # Capture output to test-output.log while also displaying it +# Example for UI tests: pwsh .github/scripts/BuildAndRunHostApp.ps1 -Platform -TestFilter "" *>&1 | Tee-Object -FilePath "$OUTPUT_DIR/test-output.log" + +# Example for device tests: +pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project -Platform -TestFilter "" *>&1 | Tee-Object -FilePath "$OUTPUT_DIR/test-output.log" ``` **Testing Loop (Iterate until SUCCESS or exhausted):** diff --git a/.github/skills/try-fix/references/example-invocation.md b/.github/skills/try-fix/references/example-invocation.md index 476cfe39605c..112f8b7ce069 100644 --- a/.github/skills/try-fix/references/example-invocation.md +++ b/.github/skills/try-fix/references/example-invocation.md @@ -8,6 +8,8 @@ problem: | test_command: | pwsh .github/scripts/BuildAndRunHostApp.ps1 -Platform android -TestFilter "Issue54321" + # For device tests use: pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=CollectionView" + # For unit tests use: dotnet test --filter "TestClassName" target_files: - src/Controls/src/Core/Handlers/Items/ItemsViewHandler.Android.cs diff --git a/.github/skills/verify-tests-fail-without-fix/SKILL.md b/.github/skills/verify-tests-fail-without-fix/SKILL.md index 7ee8c92164d2..db8bc30d93c6 100644 --- a/.github/skills/verify-tests-fail-without-fix/SKILL.md +++ b/.github/skills/verify-tests-fail-without-fix/SKILL.md @@ -1,15 +1,28 @@ --- name: verify-tests-fail-without-fix -description: Verifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation). +description: Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation). metadata: author: dotnet-maui - version: "1.0" + version: "2.0" compatibility: Requires git, PowerShell, and .NET SDK for building and running tests. --- # Verify Tests Fail Without Fix -Verifies UI tests actually catch the issue. Supports two workflow modes: +Verifies tests actually catch the issue. Supports **all test types** (UI tests, unit tests, XAML tests, device tests) and two workflow modes. + +## Supported Test Types + +| Test Type | Auto-Detected From | Runner | +|-----------|-------------------|--------| +| **UITest** | `TestCases.Shared.Tests/`, `TestCases.HostApp/` | `BuildAndRunHostApp.ps1` | +| **DeviceTest** | `DeviceTests/` | `Run-DeviceTests.ps1` | +| **UnitTest** | `*.UnitTests/`, `Graphics.Tests/` | `dotnet test` | +| **XamlUnitTest** | `Xaml.UnitTests/` | `dotnet test` | + +Test type is **auto-detected** from changed files. Override with `-TestType` if needed. + +**`-Platform` is always required.** It selects which platform to verify the fix on. ## Activation Guard @@ -64,11 +77,11 @@ Use when **creating tests before writing a fix**: - Perfect for test-first development ```bash -# Auto-detect test filter from changed test files +# Auto-detect test type and filter pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -# With explicit test filter -pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" +# Explicit test type + filter +pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -TestType UnitTest -TestFilter "Maui12345" ``` ## Mode 2: Full Verification (Fix Validation) @@ -124,43 +137,30 @@ The script auto-detects which mode to use based on whether fix files are present **Verify Failure Only Mode (no fix files):** 1. Fetches base branch from origin (if available) -2. Auto-detects test classes from changed test files -3. Runs tests (should FAIL to prove they catch the bug) -4. **Updates PR labels** based on result -5. Reports result +2. Auto-detects test type from changed files (UITest, UnitTest, XamlUnitTest, DeviceTest) +3. Auto-detects test classes from changed test files +4. Routes to the appropriate test runner +5. Runs tests (should FAIL to prove they catch the bug) +6. Reports result **Full Verification Mode (fix files detected):** 1. Fetches base branch from origin to ensure accurate diff 2. Auto-detects fix files (non-test code) from git diff -3. Auto-detects test classes from `TestCases.Shared.Tests/*.cs` +3. Auto-detects test type and test classes from changed files 4. Reverts fix files to base branch -5. Runs tests (should FAIL without fix) +5. Runs tests using the appropriate runner (should FAIL without fix) 6. Restores fix files -7. Runs tests (should PASS with fix) +7. Runs tests using the appropriate runner (should PASS with fix) 8. **Generates markdown reports**: - `CustomAgentLogsTmp/TestValidation/verification-report.md` - Full detailed report - `CustomAgentLogsTmp/PRState/verification-report.md` - Validate section for agent -9. **Updates PR labels** based on result -10. Reports result - -## PR Labels +9. Reports result -The skill automatically manages two labels on the PR to indicate verification status: - -| Label | Color | When Applied | -|-------|-------|--------------| -| `s/ai-reproduction-confirmed` | 🟢 Green (#2E7D32) | Tests correctly FAIL without fix (AI verified tests catch the bug) | -| `s/ai-reproduction-failed` | 🟠 Orange (#E65100) | Tests PASS without fix (AI verified tests don't catch the bug) | - -**Behavior:** -- When verification passes, adds `s/ai-reproduction-confirmed` and removes `s/ai-reproduction-failed` if present -- When verification fails, adds `s/ai-reproduction-failed` and removes `s/ai-reproduction-confirmed` if present -- If a PR is re-verified after fixing tests, labels are updated accordingly -- No label = AI hasn't verified tests yet +**Note:** PR label management (`s/ai-reproduction-confirmed` / `s/ai-reproduction-failed`) is handled by `Review-PR.ps1`, not by this script. ## Output Files -The skill generates output files under `CustomAgentLogsTmp/PRState//verify-tests-fail/`: +The skill generates output files under `CustomAgentLogsTmp/PRState//PRAgent/gate/verify-tests-fail/`: | File | Description | |------|-------------| @@ -169,19 +169,26 @@ The skill generates output files under `CustomAgentLogsTmp/PRState//ve | `test-without-fix.log` | Full test output from run without fix | | `test-with-fix.log` | Full test output from run with fix | -**Plus UI test logs in** `CustomAgentLogsTmp/UITests/`: -- `android-device.log` or `ios-device.log` - Device logs -- `test-output.log` - NUnit test output +**Plus test logs in** `CustomAgentLogsTmp/`: +- `UITests/` - UI test device logs and output +- `DeviceTests/` - Device test output +- `UnitTests/` - Unit test output **Example structure:** ``` CustomAgentLogsTmp/ -├── UITests/ # Shared UI test logs +├── UITests/ # UI test logs │ ├── android-device.log │ └── test-output.log +├── DeviceTests/ # Device test logs +│ └── test-output.log +├── UnitTests/ # Unit/XAML test logs +│ └── test-output.log └── PRState/ └── 27847/ - └── verify-tests-fail/ + └── PRAgent/ + └── gate/ + └── verify-tests-fail/ ├── verification-report.md # Full detailed report ├── verification-log.txt ├── test-without-fix.log @@ -210,6 +217,9 @@ CustomAgentLogsTmp/ # Require full verification (fail if no fix files detected) - recommended -RequireFullVerification +# Explicit test type (auto-detected if omitted) +-TestType UnitTest # or XamlUnitTest, DeviceTest, UITest + # Explicit test filter -TestFilter "Issue32030|ButtonUITests" diff --git a/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 b/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 index 5eacfed3135a..54fff79e395f 100644 --- a/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 +++ b/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 @@ -1,7 +1,7 @@ #!/usr/bin/env pwsh <# .SYNOPSIS - Verifies that UI tests catch the bug. Supports two modes: verify failure only or full verification. + Verifies that tests catch the bug. Supports all test types and two verification modes. .DESCRIPTION This script verifies that tests actually catch the issue. It supports two modes: @@ -19,8 +19,19 @@ The script auto-detects which mode to use based on whether fix files are present. Fix files and test filters are auto-detected from the git diff (non-test files that changed). + SUPPORTED TEST TYPES (auto-detected from changed files): + - UITest: Appium UI tests (TestCases.HostApp / TestCases.Shared.Tests) + - UnitTest: xUnit unit tests (*.UnitTests projects) + - XamlUnitTest: XAML unit tests (Xaml.UnitTests) + - DeviceTest: Device tests (*.DeviceTests projects) + .PARAMETER Platform Target platform: "android", "ios", "catalyst" (MacCatalyst), or "windows" + Required for all test types. + +.PARAMETER TestType + Explicit test type override. If not provided, auto-detected from changed files. + Valid values: UITest, UnitTest, XamlUnitTest, DeviceTest .PARAMETER TestFilter Test filter to pass to dotnet test (e.g., "FullyQualifiedName~Issue12345"). @@ -33,34 +44,35 @@ .PARAMETER BaseBranch Branch to revert files from. Auto-detected from PR if not specified. -.PARAMETER OutputDir - Directory to store results (default: "CustomAgentLogsTmp/TestValidation") - .PARAMETER RequireFullVerification If set, the script will fail if it cannot run full verification mode (i.e., if no fix files are detected). Without this flag, the script will automatically run in verify failure only mode when no fix files are found. .EXAMPLE - # Verify failure only mode - tests should fail (test creation workflow) + # Auto-detect everything (test type, filter, platform) ./verify-tests-fail.ps1 -Platform android .EXAMPLE - # Full verification mode - require fix files to be present - ./verify-tests-fail.ps1 -Platform android -RequireFullVerification + # Verify unit tests (no platform needed) + ./verify-tests-fail.ps1 -TestType UnitTest -TestFilter "Maui12345" .EXAMPLE - # Specify test filter explicitly (works in both modes) - ./verify-tests-fail.ps1 -Platform android -TestFilter "Issue32030" + # Verify XAML unit tests + ./verify-tests-fail.ps1 -TestType XamlUnitTest + +.EXAMPLE + # Full verification mode for UI tests + ./verify-tests-fail.ps1 -Platform android -RequireFullVerification .EXAMPLE # Specify everything explicitly - ./verify-tests-fail.ps1 -Platform ios -TestFilter "Issue12345" ` + ./verify-tests-fail.ps1 -Platform ios -TestType UITest -TestFilter "Issue12345" ` -FixFiles @("src/Controls/src/Core/SomeFile.cs") #> param( - [Parameter(Mandatory = $true)] + [Parameter(Mandatory = $false)] [ValidateSet("android", "ios", "catalyst", "maccatalyst", "windows")] [string]$Platform, @@ -77,7 +89,11 @@ param( [string]$PRNumber, [Parameter(Mandatory = $false)] - [switch]$RequireFullVerification + [switch]$RequireFullVerification, + + [Parameter(Mandatory = $false)] + [ValidateSet("UITest", "UnitTest", "XamlUnitTest", "DeviceTest")] + [string]$TestType ) $ErrorActionPreference = "Stop" @@ -88,6 +104,12 @@ if ($Platform -eq "maccatalyst") { $Platform = "catalyst" } +# Platform is required for UI and device tests, optional for unit/XAML tests +if ($TestType -in @("UITest", "DeviceTest") -and -not $Platform) { + Write-Error "$TestType requires -Platform parameter (android, ios, catalyst, windows)." + exit 1 +} + # ============================================================ # Detect PR number if not provided # ============================================================ @@ -144,72 +166,628 @@ $BaselineScript = Join-Path $RepoRoot ".github/scripts/EstablishBrokenBaseline.p # Import Test-IsTestFile and Find-MergeBase from shared script . $BaselineScript +# Import the shared test detection script +$DetectTestsScript = Join-Path $RepoRoot ".github/scripts/shared/Detect-TestsInDiff.ps1" + # ============================================================ -# Auto-detect test filter from changed files +# Test type detection from changed files # ============================================================ -function Get-AutoDetectedTestFilter { - param([string]$MergeBase) - $changedFiles = @() - if ($MergeBase) { - $changedFiles = git diff $MergeBase HEAD --name-only 2>$null +# Maps file path patterns to test types +$script:TestTypePatterns = @( + @{ Pattern = "TestCases\.(Shared\.Tests|HostApp|Android\.Tests|iOS\.Tests|Mac\.Tests|WinUI\.Tests)"; Type = "UITest" } + @{ Pattern = "Xaml\.UnitTests/"; Type = "XamlUnitTest" } + @{ Pattern = "DeviceTests/"; Type = "DeviceTest" } + @{ Pattern = "(? + param([string[]]$ChangedFiles) + + $result = @{ + Type = $null + TestFiles = @() + Project = $null + ProjectPath = $null } - # If no merge-base, use git status to find unstaged/staged files - if (-not $changedFiles -or $changedFiles.Count -eq 0) { - $changedFiles = git diff --name-only 2>$null - if (-not $changedFiles -or $changedFiles.Count -eq 0) { - $changedFiles = git diff --cached --name-only 2>$null + foreach ($file in $ChangedFiles) { + if ($file -notmatch "\.cs$" -and $file -notmatch "\.xaml$") { continue } + + foreach ($mapping in $script:TestTypePatterns) { + if ($file -match $mapping.Pattern) { + $result.TestFiles += $file + + # First match wins for type (priority order in $TestTypePatterns) + if (-not $result.Type) { + $result.Type = $mapping.Type + } elseif ($result.Type -ne $mapping.Type) { + # Multiple test types detected — warn and keep the first (highest priority) + Write-Host "⚠️ Multiple test types detected ($($result.Type) and $($mapping.Type)). Using $($result.Type)." -ForegroundColor Yellow + Write-Host " To override, use -TestType parameter explicitly." -ForegroundColor Yellow + continue + } + + # Detect specific project for unit tests + if ($mapping.Type -eq "UnitTest") { + foreach ($projName in $script:UnitTestProjectMap.Keys) { + if ($file -match [regex]::Escape($projName) -or $file -match ($projName -replace '\.', '/')) { + $result.Project = $projName + $result.ProjectPath = $script:UnitTestProjectMap[$projName] + } + } + # Fallback: infer project from directory structure + if (-not $result.Project) { + foreach ($projName in $script:UnitTestProjectMap.Keys) { + $projDir = Split-Path $script:UnitTestProjectMap[$projName] + if ($file -like "$projDir/*") { + $result.Project = $projName + $result.ProjectPath = $script:UnitTestProjectMap[$projName] + break + } + } + } + } + + # Detect specific project for device tests + if ($mapping.Type -eq "DeviceTest") { + foreach ($projName in $script:DeviceTestProjectMap.Keys) { + $projNamePattern = $projName -replace '\.', '[\./]' + if ($file -match $projNamePattern) { + $result.Project = $script:DeviceTestProjectMap[$projName] + break + } + } + } + + break # file matched a pattern, move to next file + } } } - # Find test files (files in test directories that are .cs files) - $testFiles = @() - foreach ($file in $changedFiles) { - if ($file -match "TestCases\.(Shared\.Tests|HostApp).*\.cs$" -and $file -notmatch "^_") { - $testFiles += $file + return $result +} + +# ============================================================ +# Run tests based on detected type +# ============================================================ +function Invoke-TestRun { + <# + .SYNOPSIS + Runs tests using the appropriate runner for the detected test type. + .DESCRIPTION + Routes to BuildAndRunHostApp.ps1 for UI tests, dotnet test for unit/XAML tests, + or Run-DeviceTests.ps1 for device tests. Uses Start-Emulator.ps1 for consistent + device booting across all test types that need a platform. + .OUTPUTS + Returns the path to the test output log file. + #> + param( + [string]$DetectedTestType, + [string]$Filter, + [string]$DetectedProject, + [string]$DetectedProjectPath, + [string]$LogFile + ) + + # Boot device/simulator once for test types that need a platform. + # Both BuildAndRunHostApp.ps1 and Run-DeviceTests.ps1 use Start-Emulator.ps1 + # internally, but we pre-boot here to ensure a consistent UDID is shared + # across multiple test runs in the same gate session. + if ($DetectedTestType -in @("UITest", "DeviceTest") -and -not $script:BootedDeviceUdid) { + if (-not $Platform) { + Write-Host "❌ $DetectedTestType tests require -Platform (android, ios, catalyst, windows)" -ForegroundColor Red + exit 1 + } + + # catalyst/maccatalyst/windows run on host — no emulator needed + $emulatorPlatform = switch ($Platform) { + "catalyst" { $null } + "windows" { $null } + default { $Platform } + } + + if ($emulatorPlatform) { + if ($DeviceUdid) { + $script:BootedDeviceUdid = $DeviceUdid + } else { + Write-Host "🔹 Booting $Platform device/simulator (shared across all test runs)..." -ForegroundColor Cyan + $startEmulatorScript = Join-Path $RepoRoot ".github/scripts/shared/Start-Emulator.ps1" + $emulatorParams = @{ Platform = $emulatorPlatform } + $script:BootedDeviceUdid = & $startEmulatorScript @emulatorParams + if ($LASTEXITCODE -ne 0) { + Write-Host "❌ Failed to boot device" -ForegroundColor Red + exit 1 + } + } + Write-Host "✅ Device ready: $($script:BootedDeviceUdid)" -ForegroundColor Green + } else { + $script:BootedDeviceUdid = "host" } } - if ($testFiles.Count -eq 0) { - return $null + switch ($DetectedTestType) { + "UITest" { + if (-not $Platform) { + Write-Host "❌ UI tests require -Platform (android, ios, catalyst, windows)" -ForegroundColor Red + exit 1 + } + $buildScript = Join-Path $RepoRoot ".github/scripts/BuildAndRunHostApp.ps1" + $uiParams = @{ + Platform = $Platform + TestFilter = $Filter + Rebuild = $true + } + if ($script:BootedDeviceUdid -and $script:BootedDeviceUdid -ne "host") { + $uiParams.DeviceUdid = $script:BootedDeviceUdid + } + # Capture all output — includes build, deploy, and test results + $scriptOutput = & $buildScript @uiParams 2>&1 + $scriptOutput | Out-File -FilePath $LogFile -Force -Encoding utf8 + return $LogFile + } + + "XamlUnitTest" { + $projectPath = Join-Path $RepoRoot "src/Controls/tests/Xaml.UnitTests/Controls.Xaml.UnitTests.csproj" + Write-Host "🧪 Running XAML unit tests: $projectPath" -ForegroundColor Cyan + Write-Host " Filter: $Filter" -ForegroundColor Gray + + $testOutputFile = Join-Path $RepoRoot "CustomAgentLogsTmp/UnitTests/test-output.log" + $testOutputDir = Split-Path $testOutputFile + if (-not (Test-Path $testOutputDir)) { + New-Item -ItemType Directory -Force -Path $testOutputDir | Out-Null + } + + $testArgs = @( + "test", $projectPath, + "--configuration", "Debug", + "--logger", "console;verbosity=normal" + ) + if ($Filter) { + $testArgs += @("--filter", $Filter) + } + + $scriptOutput = & dotnet @testArgs 2>&1 + $scriptOutput | Out-File -FilePath $LogFile -Force -Encoding utf8 + return $LogFile + } + + "UnitTest" { + $projectPath = if ($DetectedProjectPath) { + Join-Path $RepoRoot $DetectedProjectPath + } else { + # Fallback: try to find project from filter + $null + } + + if (-not $projectPath -or -not (Test-Path $projectPath)) { + Write-Host "❌ Could not determine unit test project to run." -ForegroundColor Red + Write-Host " Detected project: $DetectedProject" -ForegroundColor Yellow + Write-Host " Path: $projectPath" -ForegroundColor Yellow + exit 1 + } + + Write-Host "🧪 Running unit tests: $projectPath" -ForegroundColor Cyan + Write-Host " Filter: $Filter" -ForegroundColor Gray + + $testOutputFile = Join-Path $RepoRoot "CustomAgentLogsTmp/UnitTests/test-output.log" + $testOutputDir = Split-Path $testOutputFile + if (-not (Test-Path $testOutputDir)) { + New-Item -ItemType Directory -Force -Path $testOutputDir | Out-Null + } + + $testArgs = @( + "test", $projectPath, + "--configuration", "Debug", + "--logger", "console;verbosity=normal" + ) + if ($Filter) { + $testArgs += @("--filter", $Filter) + } + + $scriptOutput = & dotnet @testArgs 2>&1 + $scriptOutput | Out-File -FilePath $LogFile -Force -Encoding utf8 + return $LogFile + } + + "DeviceTest" { + if (-not $Platform) { + Write-Host "❌ Device tests require -Platform (android, ios, maccatalyst, windows)" -ForegroundColor Red + exit 1 + } + + $devicePlatform = if ($Platform -eq "catalyst") { "maccatalyst" } else { $Platform } + $deviceProject = if ($DetectedProject) { $DetectedProject } else { "Controls" } + + $deviceTestScript = Join-Path $RepoRoot ".github/skills/run-device-tests/scripts/Run-DeviceTests.ps1" + Write-Host "🧪 Running device tests: $deviceProject on $devicePlatform" -ForegroundColor Cyan + Write-Host " Filter: $Filter" -ForegroundColor Gray + + $testOutputFile = Join-Path $RepoRoot "CustomAgentLogsTmp/DeviceTests/test-output.log" + $testOutputDir = Split-Path $testOutputFile + if (-not (Test-Path $testOutputDir)) { + New-Item -ItemType Directory -Force -Path $testOutputDir | Out-Null + } + + $deviceParams = @{ + Project = $deviceProject + Platform = $devicePlatform + Configuration = "Release" + } + + # Pass filter through — detection ensures it's Category= format + if ($Filter) { + $deviceParams.TestFilter = $Filter + } + + if ($script:BootedDeviceUdid -and $script:BootedDeviceUdid -ne "host") { + $deviceParams.DeviceUdid = $script:BootedDeviceUdid + } + + $scriptOutput = & $deviceTestScript @deviceParams 2>&1 + $scriptOutput | Out-File -FilePath $LogFile -Force -Encoding utf8 + return $LogFile + } + + default { + Write-Host "❌ Unknown test type: $DetectedTestType" -ForegroundColor Red + exit 1 + } } +} - # Extract class names from test files - $testClassNames = @() - foreach ($file in $testFiles) { - if ($file -match "TestCases\.Shared\.Tests.*\.cs$") { - $fullPath = Join-Path $RepoRoot $file - if (Test-Path $fullPath) { - $content = Get-Content $fullPath -Raw - if ($content -match "public\s+(partial\s+)?class\s+(\w+)") { - $className = $matches[2] - if ($className -notmatch "^_" -and $testClassNames -notcontains $className) { - $testClassNames += $className - } +# ============================================================ +# Run test with retry on environment errors +# ============================================================ +function Invoke-TestRunWithRetry { + param( + [hashtable]$TestEntry, + [string]$LogFile, + [int]$MaxRetries = 3 + ) + + for ($attempt = 1; $attempt -le $MaxRetries; $attempt++) { + $logFileAttempt = if ($attempt -gt 1) { "$LogFile.attempt$attempt" } else { $LogFile } + + # Clear stale test output files before each run to prevent + # reading results from the previous run + $staleOutputPaths = @( + (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log"), + (Join-Path $RepoRoot "CustomAgentLogsTmp/DeviceTests/test-output.log"), + (Join-Path $RepoRoot "CustomAgentLogsTmp/UnitTests/test-output.log") + ) + foreach ($stale in $staleOutputPaths) { + if (Test-Path $stale) { Remove-Item $stale -Force } + } + + $testOutputLog = Invoke-TestRun ` + -DetectedTestType $TestEntry.Type ` + -Filter $TestEntry.Filter ` + -DetectedProject $TestEntry.Project ` + -DetectedProjectPath $TestEntry.ProjectPath ` + -LogFile $logFileAttempt + + $result = Get-TestResultFromOutput -LogFile $testOutputLog -TestFilter $TestEntry.Filter + + if (-not $result.EnvError) { + return $result + } + + if ($attempt -lt $MaxRetries) { + Write-Host " ⚠️ Environment error (attempt $attempt/$MaxRetries): $($result.Error) — retrying in 30s..." -ForegroundColor Yellow + + # On app launch failures, reboot the simulator/emulator to recover + if ($result.Error -match "APP_LAUNCH_FAILURE|exit code.*83|app.*crash" -and $script:BootedDeviceUdid -and $script:BootedDeviceUdid -ne "host") { + Write-Host " 🔄 Rebooting device ($($script:BootedDeviceUdid)) to recover from app launch failure..." -ForegroundColor Yellow + if ($Platform -in @("ios", "catalyst", "maccatalyst")) { + xcrun simctl shutdown $script:BootedDeviceUdid 2>$null + Start-Sleep -Seconds 5 + xcrun simctl boot $script:BootedDeviceUdid 2>$null + } elseif ($Platform -eq "android") { + adb -s $script:BootedDeviceUdid reboot 2>$null } } + + Start-Sleep -Seconds 30 + } else { + Write-Host " ⚠️ Environment error persisted after $MaxRetries attempts: $($result.Error)" -ForegroundColor Yellow + return $result } } +} - # Fallback: use file names without extension - if ($testClassNames.Count -eq 0) { - foreach ($file in $testFiles) { - $fileName = [System.IO.Path]::GetFileNameWithoutExtension($file) - if ($fileName -notmatch "^_" -and $testClassNames -notcontains $fileName) { - $testClassNames += $fileName +# ============================================================ +# Parse test results from output (supports all test types) +# ============================================================ +function Get-TestResultFromOutput { + <# + .SYNOPSIS + Parses test results from a log file. Supports dotnet test, BuildAndRunHostApp, + and device test (xharness) output formats. + .DESCRIPTION + When TestFilter is provided and the log contains device test output with + [PASS]/[FAIL] markers, checks only whether the specific filtered test(s) + passed — ignoring unrelated test failures. + .OUTPUTS + Hashtable with keys: Passed (bool), Total, PassCount (alias: Passed count), + FailCount (alias: Failed count), Skipped, Error, FailureReason + #> + param( + [string]$LogFile, + [string]$TestFilter + ) + + if (-not (Test-Path $LogFile)) { + return @{ Passed = $false; Error = "Test output log not found: $LogFile"; Total = 0; Failed = 0; Skipped = 0 } + } + + $content = Get-Content $LogFile -Raw + + # ── First, check if tests actually ran and produced results ── + # This must come BEFORE env error checks because xharness can report + # exit code 83 (APP_LAUNCH_FAILURE) even when tests ran successfully + # (e.g., due to cleanup/teardown issues after test completion). + + # Device test output: check Passed/Failed counts from Run-DeviceTests.ps1 + # Format: " Passed: 57\n Failed: 0" + # Run-DeviceTests.ps1 may retry internally, producing multiple Passed:/Failed: blocks. + # Use the MAXIMUM Passed count (the successful run), not the last one (which may be a crash with 0/0). + $allPassMatches = [regex]::Matches($content, "(?m)^\s*Passed:\s*(\d+)") + $allFailMatches = [regex]::Matches($content, "(?m)^\s*Failed:\s*(\d+)") + + if ($allPassMatches.Count -gt 0) { + $devicePassCount = ($allPassMatches | ForEach-Object { [int]$_.Groups[1].Value } | Measure-Object -Maximum).Maximum + $deviceFailCount = if ($allFailMatches.Count -gt 0) { ($allFailMatches | ForEach-Object { [int]$_.Groups[1].Value } | Measure-Object -Maximum).Maximum } else { 0 } + $deviceTotal = $devicePassCount + $deviceFailCount + + Write-Host " 📊 Parsed test results: Passed=$devicePassCount Failed=$deviceFailCount Total=$deviceTotal (from $($allPassMatches.Count) result blocks)" -ForegroundColor Gray + + # If tests actually ran (passed > 0), trust the results over exit codes + if ($devicePassCount -gt 0) { + if ($deviceFailCount -gt 0) { + return @{ + Passed = $false; FailCount = $deviceFailCount; Failed = $deviceFailCount + PassCount = $devicePassCount; Total = $deviceTotal; Skipped = 0 + FailureReason = "Device tests: $deviceFailCount of $deviceTotal failed" + } + } + return @{ + Passed = $true; PassCount = $devicePassCount; Failed = 0 + FailCount = 0; Total = $deviceTotal; Skipped = 0 } } } - if ($testClassNames.Count -eq 0) { + # ── Environment/infrastructure errors (only if no real test results above) ── + $envErrorPatterns = @( + @{ Pattern = "error ADB0010.*InstallFailedException"; Message = "App install failed (ADB broken pipe)" } + @{ Pattern = "XHarness exit code:\s*83"; Message = "App failed to launch (XHarness exit 83)" } + @{ Pattern = "Application test run crashed"; Message = "App crashed during test run" } + @{ Pattern = "SIGABRT.*load_aot_module"; Message = "App crashed during AOT loading" } + @{ Pattern = "AppiumServerHasNotBeenStartedLocally"; Message = "Appium server failed to start" } + @{ Pattern = "no such element.*could not be located"; Message = "Test element not found (app may not have loaded)" } + ) + foreach ($envErr in $envErrorPatterns) { + if ($content -match $envErr.Pattern) { + return @{ Passed = $false; EnvError = $true; Error = $envErr.Message; FailCount = 0; Failed = 0; Total = 0; Skipped = 0 } + } + } + + # Check for build failures (before any test results) + if ($content -match "Build FAILED" -or $content -match "Build failed with exit code" -or $content -match "error MSB\d+" -or $content -match "error CS\d+") { + return @{ Passed = $false; Error = "Build failed before tests could run"; FailCount = 0; Failed = 0; Total = 0; Skipped = 0 } + } + + # --- Device test output: [PASS]/[FAIL] markers from xharness --- + # When TestFilter is specified and the log contains device test markers, + # check only the filtered test results. Device tests run ALL tests regardless + # of filter, so unrelated failures must be ignored. + if ($TestFilter -and $content -match '\[PASS\]|\[FAIL\]') { + $filterNames = $TestFilter -split '\|' + $passedTests = @() + $failedTests = @() + + foreach ($name in $filterNames) { + $name = $name.Trim() + if (-not $name) { continue } + # Match lines like: [PASS] Share_MultipleFilesIntent_HasClipData + # [FAIL] Share_MultipleFilesIntent_HasClipData + if ($content -match "\[PASS\]\s+$([regex]::Escape($name))\b") { + $passedTests += $name + } + elseif ($content -match "\[FAIL\]\s+$([regex]::Escape($name))\b") { + $failedTests += $name + } + } + + $totalFound = $passedTests.Count + $failedTests.Count + if ($totalFound -gt 0) { + if ($failedTests.Count -gt 0) { + return @{ + Passed = $false + FailCount = $failedTests.Count + Failed = $failedTests.Count + PassCount = $passedTests.Count + Total = $totalFound + Skipped = 0 + FailureReason = "Filtered test(s) failed: $($failedTests -join ', ')" + } + } + return @{ + Passed = $true + PassCount = $passedTests.Count + Failed = 0 + FailCount = 0 + Total = $totalFound + Skipped = 0 + } + } + # Filter specified but tests not found in output — fall through to general parsing + } + + # --- dotnet test output --- + # Check for "Test Run Failed" (dotnet test) + if ($content -match "Test Run Failed") { + $failCount = 0 + $passCount = 0 + $skipped = 0 + if ($content -match "^\s+Failed:\s*(\d+)") { $failCount = [int]$matches[1] } + elseif ($content -match "Failed:\s*(\d+)") { $failCount = [int]$matches[1] } + if ($content -match "^\s+Passed:\s*(\d+)") { $passCount = [int]$matches[1] } + if ($content -match "^\s+Skipped:\s*(\d+)") { $skipped = [int]$matches[1] } + + # Extract failure details: test name, duration, error message + $failureDetails = @() + # Match: "Failed TestName [duration]" followed by "Error Message:" block + $failedTestMatches = [regex]::Matches($content, '(?m)^\s*Failed\s+(\S+)\s*\[([^\]]+)\]') + foreach ($m in $failedTestMatches) { + $failureDetails += "$($m.Groups[1].Value) [$($m.Groups[2].Value)]" + } + # Extract error messages + $errorMsgMatches = [regex]::Matches($content, '(?ms)Error Message:\s*\n\s*(.+?)(?=\n\s*Stack Trace:|\n\s*$|\n\s*\d+\))') + $errorMessages = @() + foreach ($m in $errorMsgMatches) { + $msg = $m.Groups[1].Value.Trim() + if ($msg.Length -gt 200) { $msg = $msg.Substring(0, 200) + "..." } + $errorMessages += $msg + } + + $failureReason = if ($failureDetails.Count -gt 0) { $failureDetails -join "; " } else { $null } + $failureMessage = if ($errorMessages.Count -gt 0) { $errorMessages -join "; " } else { $null } + + return @{ + Passed = $false; FailCount = $failCount; PassCount = $passCount; Failed = $failCount + Total = $failCount + $passCount + $skipped; Skipped = $skipped + FailureReason = $failureReason; FailureMessage = $failureMessage + } + } + + # Check for "Test Run Successful" (dotnet test) + if ($content -match "Test Run Successful") { + $passCount = 0 + $skipped = 0 + if ($content -match "^\s+Passed:\s*(\d+)") { $passCount = [int]$matches[1] } + elseif ($content -match "Total tests:\s*(\d+)") { $passCount = [int]$matches[1] } + if ($content -match "^\s+Skipped:\s*(\d+)") { $skipped = [int]$matches[1] } + return @{ Passed = $true; PassCount = $passCount; Failed = 0; Skipped = $skipped; Total = $passCount + $skipped } + } + + # Check for failures first - but only if count > 0 + if ($content -match "Failed:\s*(\d+)") { + $failCount = [int]$matches[1] + if ($failCount -gt 0) { + return @{ Passed = $false; FailCount = $failCount; Failed = $failCount; PassCount = 0; Total = $failCount; Skipped = 0 } + } + } + + # Check for passes + if ($content -match "Passed:\s*(\d+)") { + $passCount = [int]$matches[1] + if ($passCount -gt 0) { + return @{ Passed = $true; PassCount = $passCount; Failed = 0; Total = $passCount; Skipped = 0 } + } + } + + # Zero tests ran (Passed: 0, Failed: 0) — treat as env error, not success + if ($content -match "Passed:\s*0" -and $content -match "Failed:\s*0") { + return @{ Passed = $false; EnvError = $true; Error = "Zero tests ran (Passed: 0, Failed: 0)"; Total = 0; Failed = 0; Skipped = 0 } + } + + return @{ Passed = $false; Error = "Could not parse test results"; Total = 0; Failed = 0; Skipped = 0 } +} + + +# ============================================================ +# Auto-detect tests from changed files using shared detection +# ============================================================ +function Get-AutoDetectedTests { + <# + .SYNOPSIS + Detects all tests in the current diff using the shared Detect-TestsInDiff.ps1 script. + .OUTPUTS + Array of test group hashtables from Detect-TestsInDiff.ps1 + #> + param([string]$MergeBase) + + $params = @{} + + # Prefer PR number (GitHub API gives exact PR files, not polluted by branch diff) + if ($PRNumber) { + $params.PRNumber = $PRNumber + } elseif ($MergeBase) { + $changedFiles = git diff $MergeBase HEAD --name-only 2>$null + if (-not $changedFiles -or $changedFiles.Count -eq 0) { + $changedFiles = git diff --name-only 2>$null + if (-not $changedFiles -or $changedFiles.Count -eq 0) { + $changedFiles = git diff --cached --name-only 2>$null + } + } + if ($changedFiles) { + $params.ChangedFiles = $changedFiles + } + } + + # Fall back to PR number if no changed files from git diff + if (-not $params.ContainsKey("ChangedFiles") -and $PRNumber) { + $params.PRNumber = $PRNumber + } + + $results = & $DetectTestsScript @params 6>$null + return $results +} + +# Keep the old function for backward compatibility but delegate to new detection +function Get-AutoDetectedTestFilter { + param([string]$MergeBase) + + $tests = Get-AutoDetectedTests -MergeBase $MergeBase + if (-not $tests -or $tests.Count -eq 0) { return $null } + # Return the first test's info for single-test backward compatibility + $first = $tests[0] return @{ - Filter = if ($testClassNames.Count -eq 1) { $testClassNames[0] } else { $testClassNames -join "|" } - ClassNames = $testClassNames + Filter = $first.Filter + ClassNames = @($first.TestName) + TestType = $first.Type + Project = $first.Project + ProjectPath = $first.ProjectPath + AllTests = $tests } } @@ -337,24 +915,37 @@ if ($DetectedFixFiles.Count -eq 0) { Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Cyan Write-Host "" - # Auto-detect test filter if not provided + # Auto-detect tests if filter not provided + $AllDetectedTests = @() + if (-not $TestFilter) { Write-Host "🔍 Auto-detecting test filter from changed test files..." -ForegroundColor Cyan $filterResult = Get-AutoDetectedTestFilter -MergeBase $MergeBase if (-not $filterResult) { - Write-Host "❌ Could not auto-detect test filter. No test files found in changed files." -ForegroundColor Red - Write-Host " Looking for files matching: TestCases.(Shared.Tests|HostApp)/*.cs" -ForegroundColor Yellow - Write-Host " Please provide -TestFilter parameter explicitly." -ForegroundColor Yellow - exit 1 + Write-Host "⚠️ No tests detected in this PR." -ForegroundColor Yellow + Write-Host " Searched for: UI tests, unit tests, XAML tests, device tests" -ForegroundColor Yellow + Write-Host " Consider adding tests via write-tests-agent." -ForegroundColor Cyan + # Exit code 2 = no tests found (distinct from 1 = verification failed) + exit 2 } - $TestFilter = $filterResult.Filter - Write-Host "✅ Auto-detected $($filterResult.ClassNames.Count) test class(es):" -ForegroundColor Green - foreach ($name in $filterResult.ClassNames) { - Write-Host " - $name" -ForegroundColor White + $AllDetectedTests = @($filterResult.AllTests) + + Write-Host "✅ Auto-detected $($AllDetectedTests.Count) test(s):" -ForegroundColor Green + foreach ($t in $AllDetectedTests) { + $icon = switch ($t.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "❓" } } + Write-Host " $icon [$($t.Type)] $($t.TestName) (filter: $($t.Filter))" -ForegroundColor White } - Write-Host " Filter: $TestFilter" -ForegroundColor Cyan + } else { + $effectiveType = if ($TestType) { $TestType } else { "UITest" } + $AllDetectedTests = @(@{ + Type = $effectiveType + TestName = $TestFilter + Filter = $TestFilter + Project = $null + ProjectPath = $null + }) } # Create output directory @@ -362,28 +953,39 @@ if ($DetectedFixFiles.Count -eq 0) { New-Item -ItemType Directory -Force -Path $OutputPath | Out-Null $ValidationLog = Join-Path $OutputPath "verification-log.txt" - $TestLog = Join-Path $OutputPath "test-failure-verification.log" # Initialize log "" | Set-Content $ValidationLog "=========================================" | Add-Content $ValidationLog "Verify Tests Fail (Failure Only Mode)" | Add-Content $ValidationLog "=========================================" | Add-Content $ValidationLog + "Tests: $($AllDetectedTests.Count)" | Add-Content $ValidationLog "Platform: $Platform" | Add-Content $ValidationLog - "TestFilter: $TestFilter" | Add-Content $ValidationLog "MergeBase: $MergeBase" | Add-Content $ValidationLog "" | Add-Content $ValidationLog - Write-Host "🧪 Running tests (expecting them to FAIL)..." -ForegroundColor Cyan + Write-Host "" + Write-Host "🧪 Running $($AllDetectedTests.Count) test(s) (expecting them to FAIL)..." -ForegroundColor Cyan Write-Host "" - # Use shared BuildAndRunHostApp.ps1 infrastructure with -Rebuild to ensure clean builds - $buildScript = Join-Path $RepoRoot ".github/scripts/BuildAndRunHostApp.ps1" - & $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $TestLog - - # Parse test results using shared function - $testOutputLog = Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log" - $testResult = Get-TestResultFromLog -LogFile $testOutputLog + # Run ALL detected tests + $allResults = @() + $testIndex = 0 + foreach ($testEntry in $AllDetectedTests) { + $testIndex++ + $icon = switch ($testEntry.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "❓" } } + Write-Host "─────────────────────────────────────────────────" -ForegroundColor DarkGray + Write-Host "$icon Test $testIndex/$($AllDetectedTests.Count): [$($testEntry.Type)] $($testEntry.TestName)" -ForegroundColor Cyan + + $sanitizedName = ($testEntry.TestName -replace '[^a-zA-Z0-9_\-\.]', '_') + if ($sanitizedName.Length -gt 60) { $sanitizedName = $sanitizedName.Substring(0, 60) } + $TestLog = Join-Path $OutputPath "test-failure-$sanitizedName.log" + + $testResult = Invoke-TestRunWithRetry -TestEntry $testEntry -LogFile $TestLog + $testResult.TestName = $testEntry.TestName + $testResult.TestType = $testEntry.Type + $allResults += $testResult + } # Evaluate results Write-Host "" @@ -392,47 +994,46 @@ if ($DetectedFixFiles.Count -eq 0) { Write-Host "==========================================" Write-Host "" - if ($testResult.Error) { + $allFailed = ($allResults | Where-Object { $_.Passed }).Count -eq 0 + $hasErrors = ($allResults | Where-Object { $_.Error }).Count -gt 0 + + if ($hasErrors) { Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red Write-Host "║ ERROR PARSING TEST RESULTS ║" -ForegroundColor Red - Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Red - Write-Host "║ $($testResult.Error.PadRight(57)) ║" -ForegroundColor Red Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red + foreach ($r in ($allResults | Where-Object { $_.Error })) { + Write-Host " [$($r.TestType)] $($r.TestName): $($r.Error)" -ForegroundColor Red + } exit 1 } - if (-not $testResult.Passed) { - # Tests FAILED - this is what we want! + # Show per-test results + foreach ($r in $allResults) { + $icon = switch ($r.TestType) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "❓" } } + if (-not $r.Passed) { + Write-Host " $icon [$($r.TestType)] $($r.TestName): FAILED ✅ (expected)" -ForegroundColor Green + } else { + Write-Host " $icon [$($r.TestType)] $($r.TestName): PASSED ❌ (should fail!)" -ForegroundColor Red + } + } + Write-Host "" + + if ($allFailed) { Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Green Write-Host "║ VERIFICATION PASSED ✅ ║" -ForegroundColor Green Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Green - Write-Host "║ Tests FAILED as expected! ║" -ForegroundColor Green - Write-Host "║ ║" -ForegroundColor Green + Write-Host "║ All $($allResults.Count) test(s) FAILED as expected! ║" -ForegroundColor Green Write-Host "║ This proves the tests correctly reproduce the bug. ║" -ForegroundColor Green - Write-Host "║ You can now proceed to write the fix. ║" -ForegroundColor Green Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Green - Write-Host "" - Write-Host "Failed tests: $($testResult.FailCount)" -ForegroundColor Yellow exit 0 } else { - # Tests PASSED - this is bad! + $passedCount = ($allResults | Where-Object { $_.Passed }).Count Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red Write-Host "║ VERIFICATION FAILED ❌ ║" -ForegroundColor Red Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Red - Write-Host "║ Tests PASSED but they should FAIL! ║" -ForegroundColor Red - Write-Host "║ ║" -ForegroundColor Red - Write-Host "║ This means your tests don't actually reproduce the bug. ║" -ForegroundColor Red - Write-Host "║ ║" -ForegroundColor Red - Write-Host "║ Possible causes: ║" -ForegroundColor Red - Write-Host "║ 1. Test scenario doesn't match the issue description ║" -ForegroundColor Red - Write-Host "║ 2. Test assertions are wrong or too lenient ║" -ForegroundColor Red - Write-Host "║ 3. The bug was already fixed in this branch ║" -ForegroundColor Red - Write-Host "║ 4. The bug only happens in specific conditions ║" -ForegroundColor Red - Write-Host "║ ║" -ForegroundColor Red - Write-Host "║ Go back and revise your tests! ║" -ForegroundColor Red + Write-Host "║ $passedCount/$($allResults.Count) test(s) PASSED but should FAIL! ║" -ForegroundColor Red + Write-Host "║ Those tests don't reproduce the bug. Revise them! ║" -ForegroundColor Red Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red - Write-Host "" - Write-Host "Passed tests: $($testResult.PassCount)" -ForegroundColor Yellow exit 1 } } @@ -459,23 +1060,43 @@ foreach ($file in $FixFiles) { } # Auto-detect test filter from test files if not provided +$AllDetectedTests = @() + if (-not $TestFilter) { Write-Host "🔍 Auto-detecting test filter from changed test files..." -ForegroundColor Cyan $filterResult = Get-AutoDetectedTestFilter -MergeBase $MergeBase if (-not $filterResult) { - Write-Host "❌ Could not auto-detect test filter. No test files found in changed files." -ForegroundColor Red - Write-Host " Looking for files matching: TestCases.(Shared.Tests|HostApp)/*.cs" -ForegroundColor Yellow - Write-Host " Please provide -TestFilter parameter explicitly." -ForegroundColor Yellow - exit 1 + Write-Host "⚠️ No tests detected in this PR." -ForegroundColor Yellow + Write-Host " Searched for: UI tests, unit tests, XAML tests, device tests" -ForegroundColor Yellow + Write-Host " Consider adding tests via write-tests-agent." -ForegroundColor Cyan + # Exit code 2 = no tests found (distinct from 1 = verification failed) + exit 2 } - $TestFilter = $filterResult.Filter - Write-Host "✅ Auto-detected $($filterResult.ClassNames.Count) test class(es):" -ForegroundColor Green - foreach ($name in $filterResult.ClassNames) { - Write-Host " - $name" -ForegroundColor White + $AllDetectedTests = @($filterResult.AllTests) + + Write-Host "✅ Auto-detected $($AllDetectedTests.Count) test(s):" -ForegroundColor Green + foreach ($t in $AllDetectedTests) { + $icon = switch ($t.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "❓" } } + Write-Host " $icon [$($t.Type)] $($t.TestName) (filter: $($t.Filter))" -ForegroundColor White } - Write-Host " Filter: $TestFilter" -ForegroundColor Cyan +} else { + # Explicit filter provided — use single test entry with given/detected type + $effectiveType = if ($TestType) { $TestType } else { "UITest" } + $AllDetectedTests = @(@{ + Type = $effectiveType + TestName = $TestFilter + Filter = $TestFilter + Project = $null + ProjectPath = $null + Runner = switch ($effectiveType) { + "UITest" { "BuildAndRunHostApp" } + "DeviceTest" { "Run-DeviceTests" } + default { "dotnet-test" } + } + NeedsPlatform = ($effectiveType -in @("UITest", "DeviceTest")) + }) } # Create output directory @@ -501,142 +1122,178 @@ function Write-MarkdownReport { [bool]$FailedWithoutFix, [bool]$PassedWithFix, [hashtable]$WithoutFixResult, - [hashtable]$WithFixResult + [hashtable]$WithFixResult, + [array]$WithoutFixResultsList, + [array]$WithFixResultsList, + [array]$Tests, + [string]$ReportMergeBase, + [string]$ReportPlatform, + [string]$ReportBaseBranch, + [array]$ReportRevertableFiles, + [array]$ReportNewFiles ) - $reportDate = Get-Date -Format "yyyy-MM-dd HH:mm:ss" - $status = if ($VerificationPassed) { "✅ PASSED" } else { "❌ FAILED" } - $statusSymbol = if ($VerificationPassed) { "✅" } else { "❌" } + # Check for environment errors in results + $hasEnvError = ($WithoutFixResultsList | Where-Object { $_.EnvError }) -or ($WithFixResultsList | Where-Object { $_.EnvError }) - $markdown = @" -### Test Verification Report - -**Date:** $reportDate | **Platform:** $($Platform.ToUpper()) | **Status:** $status - -#### Summary - -| Check | Expected | Actual | Result | -|-------|----------|--------|--------| -| Tests WITHOUT fix | FAIL | $(if ($FailedWithoutFix) { "FAIL" } else { "PASS" }) | $(if ($FailedWithoutFix) { "✅" } else { "❌" }) | -| Tests WITH fix | PASS | $(if ($PassedWithFix) { "PASS" } else { "FAIL" }) | $(if ($PassedWithFix) { "✅" } else { "❌" }) | - -#### $statusSymbol Final Verdict - -$(if ($VerificationPassed) { - @" -**VERIFICATION PASSED** ✅ - -The tests correctly detect the issue: -- ✅ Tests **FAIL** without the fix (as expected - bug is present) -- ✅ Tests **PASS** with the fix (as expected - bug is fixed) - -**Conclusion:** The tests properly validate the fix and catch the bug when it's present. -"@ -} else { - @" -**VERIFICATION FAILED** ❌ - -$(if (-not $FailedWithoutFix) { - "❌ **Tests PASSED without fix** (should have failed)`n - The tests don't actually detect the bug`n - Tests may not be testing the right behavior`n" -})$(if (-not $PassedWithFix) { - "❌ **Tests FAILED with fix** (should have passed)`n - The fix doesn't resolve the issue`n - Tests may be broken or testing something else`n" -}) -**Possible causes:** -1. Wrong fix files specified -2. Tests don't actually test the fixed behavior -3. The issue was already fixed in base branch -4. Build caching - try clean rebuild -5. Test needs different setup or conditions -"@ -}) - ---- - -#### Configuration - -**Platform:** $Platform -**Test Filter:** $TestFilter -**Base Branch:** $BaseBranchName -**Merge Base:** $(if ($MergeBase -and $MergeBase.Length -ge 8) { $MergeBase.Substring(0, 8) } else { $MergeBase }) - -### Fix Files - -$(($RevertableFiles | ForEach-Object { "- ``$_``" }) -join "`n") - -$(if ($NewFiles.Count -gt 0) { -@" - -### New Files (Not Reverted) - -$(($NewFiles | ForEach-Object { "- ``$_``" }) -join "`n") -"@ -}) - ---- - -#### Test Results Details - -### Test Run 1: WITHOUT Fix - -**Expected:** Tests should FAIL (bug is present) -**Actual:** Tests $(if ($FailedWithoutFix) { "FAILED" } else { "PASSED" }) $(if ($FailedWithoutFix) { "✅" } else { "❌" }) - -**Test Summary:** -- Total: $($WithoutFixResult.Total) -- Passed: $($WithoutFixResult.Passed) -- Failed: $($WithoutFixResult.Failed) -- Skipped: $($WithoutFixResult.Skipped) - -$(if ($WithoutFixResult.FailureReason) { - "**Failure Reason:** ``$($WithoutFixResult.FailureReason)``" -}) - -
-View full test output (without fix) - -`````` -$(Get-Content $WithoutFixLog -Raw) -`````` - -
- ---- - -### Test Run 2: WITH Fix - -**Expected:** Tests should PASS (bug is fixed) -**Actual:** Tests $(if ($PassedWithFix) { "PASSED" } else { "FAILED" }) $(if ($PassedWithFix) { "✅" } else { "❌" }) - -**Test Summary:** -- Total: $($WithFixResult.Total) -- Passed: $($WithFixResult.Passed) -- Failed: $($WithFixResult.Failed) -- Skipped: $($WithFixResult.Skipped) - -$(if ($WithFixResult.FailureReason) { - "**Failure Reason:** ``$($WithFixResult.FailureReason)``" -}) + $status = if ($hasEnvError) { "⚠️ ENV ERROR" } elseif ($VerificationPassed) { "✅ PASSED" } else { "❌ FAILED" } + $mergeBaseShort = if ($ReportMergeBase -and $ReportMergeBase.Length -ge 8) { $ReportMergeBase.Substring(0, 8) } else { "$ReportMergeBase" } + + $lines = @() + $lines += "### Gate Result: $status" + $lines += "" + $platformDisplay = if ($ReportPlatform) { $ReportPlatform.ToUpper() } else { "N/A" } + $lines += "**Platform:** $platformDisplay · **Base:** $ReportBaseBranch · **Merge base:** ``$mergeBaseShort``" + $lines += "" + + # ── Side-by-side per-test comparison table ── + $lines += "| Test | Without Fix (expect FAIL) | With Fix (expect PASS) |" + $lines += "|------|--------------------------|------------------------|" + + foreach ($t in $Tests) { + $woResult = $WithoutFixResultsList | Where-Object { $_.TestName -eq $t.TestName } + $wResult = $WithFixResultsList | Where-Object { $_.TestName -eq $t.TestName } + + # Without fix cell + $woDur = if ($woResult.Duration) { "$([math]::Round($woResult.Duration.TotalSeconds))s" } else { "" } + if ($woResult.EnvError) { + $woCell = "⚠️ ENV ERROR" + } elseif (-not $woResult.Passed) { + $woCell = "✅ FAIL — $woDur" + } else { + $woCell = "❌ PASS — $woDur" + } -
-View full test output (with fix) + # With fix cell + $wDur = if ($wResult.Duration) { "$([math]::Round($wResult.Duration.TotalSeconds))s" } else { "" } + if ($wResult.EnvError) { + $wCell = "⚠️ ENV ERROR" + } elseif ($wResult.Passed) { + $wCell = "✅ PASS — $wDur" + } else { + $wCell = "❌ FAIL — $wDur" + } -`````` -$(Get-Content $WithFixLog -Raw) -`````` + $icon = switch ($t.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "" } } + $lines += "| $icon **$($t.TestName)** ``$($t.Filter)`` | $woCell | $wCell |" + } -
+ # ── Per-test logs (collapsible) ── + foreach ($t in $Tests) { + $sanitizedName = ($t.TestName -replace '[^a-zA-Z0-9_\-\.]', '_') + if ($sanitizedName.Length -gt 60) { $sanitizedName = $sanitizedName.Substring(0, 60) } + + $woResult = $WithoutFixResultsList | Where-Object { $_.TestName -eq $t.TestName } + $wResult = $WithFixResultsList | Where-Object { $_.TestName -eq $t.TestName } + $icon = switch ($t.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "" } } + + # Without fix log + $woLogFile = Join-Path $OutputPath "test-without-fix-$sanitizedName.log" + $woStatus = if ($woResult.EnvError) { "⚠️ ENV ERROR" } elseif (-not $woResult.Passed) { "FAIL ✅" } else { "PASS ❌" } + $woDur = if ($woResult.Duration) { " · $([math]::Round($woResult.Duration.TotalSeconds))s" } else { "" } + $lines += "" + $lines += "
" + $lines += "🔴 Without fix — $icon $($t.TestName): $woStatus$woDur" + $lines += "" + if (Test-Path $woLogFile) { + $logContent = Get-Content $woLogFile -Raw -ErrorAction SilentlyContinue + if ($logContent) { + # Truncate if too large for a PR comment (GitHub limit ~65k chars total) + if ($logContent.Length -gt 15000) { + $logContent = $logContent.Substring($logContent.Length - 15000) + $lines += "*(truncated to last 15,000 chars)*" + $lines += "" + } + $lines += '```' + $lines += $logContent + $lines += '```' + } else { + $lines += "*Log file empty*" + } + } else { + $lines += "*No log file found*" + } + $lines += "" + $lines += "
" + + # With fix log + $wLogFile = Join-Path $OutputPath "test-with-fix-$sanitizedName.log" + $wStatus = if ($wResult.EnvError) { "⚠️ ENV ERROR" } elseif ($wResult.Passed) { "PASS ✅" } else { "FAIL ❌" } + $wDur = if ($wResult.Duration) { " · $([math]::Round($wResult.Duration.TotalSeconds))s" } else { "" } + $lines += "" + $lines += "
" + $lines += "🟢 With fix — $icon $($t.TestName): $wStatus$wDur" + $lines += "" + if (Test-Path $wLogFile) { + $logContent = Get-Content $wLogFile -Raw -ErrorAction SilentlyContinue + if ($logContent) { + if ($logContent.Length -gt 15000) { + $logContent = $logContent.Substring($logContent.Length - 15000) + $lines += "*(truncated to last 15,000 chars)*" + $lines += "" + } + $lines += '```' + $lines += $logContent + $lines += '```' + } else { + $lines += "*Log file empty*" + } + } else { + $lines += "*No log file found*" + } + $lines += "" + $lines += "
" + } ---- + # ── Failure details (only if something went wrong) ── + $failureLines = @() + foreach ($r in $withoutFixResults) { + if ($r.Passed) { + $failureLines += "- ❌ **$($r.TestName)** PASSED without fix (should fail) — tests don't catch the bug" + } + if ($r.EnvError) { $failureLines += "- ⚠️ **$($r.TestName)** without fix: ``$($r.Error)``" } + } + foreach ($r in $withFixResults) { + if (-not $r.Passed -and -not $r.EnvError) { + $failureLines += "- ❌ **$($r.TestName)** FAILED with fix (should pass)" + if ($r.FailureReason) { $failureLines += " - ``$($r.FailureReason)``" } + if ($r.FailureMessage) { + $msg = if ($r.FailureMessage.Length -gt 200) { $r.FailureMessage.Substring(0, 200) + "..." } else { $r.FailureMessage } + $failureLines += " - ``$msg``" + } + } + if ($r.EnvError) { $failureLines += "- ⚠️ **$($r.TestName)** with fix: ``$($r.Error)``" } + } -#### Logs + if ($failureLines.Count -gt 0) { + $lines += "" + $lines += "
" + $lines += "⚠️ Issues found" + $lines += "" + $lines += ($failureLines -join "`n") + $lines += "" + $lines += "
" + } -- Full verification log: ``$ValidationLog`` -- Test output without fix: ``$WithoutFixLog`` -- Test output with fix: ``$WithFixLog`` -- UI test logs: ``CustomAgentLogsTmp/UITests/`` -"@ + # ── Fix files (collapsible) ── + $lines += "" + $lines += "
" + $lines += "📁 Fix files reverted ($($ReportRevertableFiles.Count) files)" + $lines += "" + foreach ($f in $ReportRevertableFiles) { + $lines += "- ``$f``" + } + if ($ReportNewFiles.Count -gt 0) { + $lines += "" + $lines += "**New files (not reverted):**" + foreach ($f in $ReportNewFiles) { + $lines += "- ``$f``" + } + } + $lines += "" + $lines += "
" - $markdown | Set-Content -Path $MarkdownReport -Encoding UTF8 + ($lines -join "`n") | Set-Content -Path $MarkdownReport -Encoding UTF8 Write-Host "" Write-Host "📄 Markdown report saved to: $MarkdownReport" -ForegroundColor Cyan } @@ -648,8 +1305,11 @@ $(Get-Content $WithFixLog -Raw) Write-Log "==========================================" Write-Log "Verify Tests Fail Without Fix" Write-Log "==========================================" +Write-Log "Tests detected: $($AllDetectedTests.Count)" +foreach ($t in $AllDetectedTests) { + Write-Log " - [$($t.Type)] $($t.TestName) (filter: $($t.Filter))" +} Write-Log "Platform: $Platform" -Write-Log "TestFilter: $TestFilter" Write-Log "FixFiles: $($FixFiles -join ', ')" Write-Log "BaseBranch: $BaseBranchName" Write-Log "MergeBase: $MergeBase" @@ -741,17 +1401,75 @@ foreach ($file in $RevertableFiles) { Write-Log " ✓ $($RevertableFiles.Count) fix file(s) reverted to merge-base state" -# Step 2: Run tests WITHOUT fix +# Step 2: Run ALL tests WITHOUT fix +Write-Host "" +Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Magenta +Write-Host "║ STEP 2: Running tests WITHOUT fix (expect FAIL) ║" -ForegroundColor Magenta +Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Magenta Write-Log "" -Write-Log "==========================================" Write-Log "STEP 2: Running tests WITHOUT fix (should FAIL)" -Write-Log "==========================================" -# Use shared BuildAndRunHostApp.ps1 infrastructure with -Rebuild to ensure clean builds -$buildScript = Join-Path $RepoRoot ".github/scripts/BuildAndRunHostApp.ps1" -& $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $WithoutFixLog +$withoutFixResults = @() +$testIndex = 0 +foreach ($testEntry in $AllDetectedTests) { + $testIndex++ + $icon = switch ($testEntry.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "❓" } } + + $sanitizedName = ($testEntry.TestName -replace '[^a-zA-Z0-9_\-\.]', '_') + if ($sanitizedName.Length -gt 60) { $sanitizedName = $sanitizedName.Substring(0, 60) } + $testLogFile = Join-Path $OutputPath "test-without-fix-$sanitizedName.log" + + # AzDO collapsible group for raw test output + Write-Host "##[group]🔴 WITHOUT FIX $testIndex/$($AllDetectedTests.Count): $icon $($testEntry.TestName) (filter: $($testEntry.Filter))" + + $sw = [System.Diagnostics.Stopwatch]::StartNew() + try { + $result = Invoke-TestRunWithRetry -TestEntry $testEntry -LogFile $testLogFile + } catch { + $result = @{ Passed = $false; Failed = 0; Total = 0; PassCount = 0; FailCount = 0; Skipped = 0; EnvError = $true; Error = $_.Exception.Message } + Write-Host " ⚠️ Test invocation threw: $($_.Exception.Message)" -ForegroundColor Yellow + } + $sw.Stop() + $result.TestName = $testEntry.TestName + $result.TestType = $testEntry.Type + $result.Duration = $sw.Elapsed + $withoutFixResults += $result + + # Print raw log inside the collapsible group so it's available but not noisy + if (Test-Path $testLogFile) { + $logLines = Get-Content $testLogFile -ErrorAction SilentlyContinue + $lineCount = if ($logLines) { $logLines.Count } else { 0 } + Write-Host " ── Log ($lineCount lines) ──" -ForegroundColor DarkGray + if ($logLines) { $logLines | ForEach-Object { Write-Host " $_" } } + } + Write-Host "##[endgroup]" + + # Print result OUTSIDE the group so it's always visible + $durStr = "$([math]::Round($sw.Elapsed.TotalSeconds))s" + $counts = if ($result.Total -gt 0) { " ($($result.Total) total, $($result.Failed) failed)" } else { "" } + if ($result.EnvError) { + Write-Host " ⚠️ $($testEntry.TestName): ENV ERROR$counts — $durStr — $($result.Error)" -ForegroundColor Yellow + } elseif (-not $result.Passed) { + Write-Host " ✅ $($testEntry.TestName): FAILED$counts — $durStr (expected)" -ForegroundColor Green + if ($result.FailureReason) { Write-Host " └─ $($result.FailureReason)" -ForegroundColor DarkGray } + } else { + Write-Host " ❌ $($testEntry.TestName): PASSED$counts — $durStr (unexpected!)" -ForegroundColor Red + } + Write-Log " [$($testEntry.Type)] $($testEntry.TestName): Passed=$($result.Passed) Failed=$($result.Failed) [$durStr]" +} + +# Combine into a single summary for backward compatibility +$withoutFixResult = @{ + Passed = ($withoutFixResults | Where-Object { $_.Passed }).Count -eq $withoutFixResults.Count + PassCount = ($withoutFixResults | Measure-Object -Property PassCount -Sum).Sum + FailCount = ($withoutFixResults | Measure-Object -Property FailCount -Sum).Sum + Failed = ($withoutFixResults | Measure-Object -Property Failed -Sum).Sum + Skipped = ($withoutFixResults | Measure-Object -Property Skipped -Sum).Sum + Total = ($withoutFixResults | Measure-Object -Property Total -Sum).Sum +} -$withoutFixResult = Get-TestResultFromLog -LogFile (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log") +# Save combined log +$withoutFixResults | ForEach-Object { "[$($_.TestType)] $($_.TestName): Passed=$($_.Passed) Failed=$($_.Failed)" } | Out-File $WithoutFixLog -Append # Step 3: Restore fix files from current branch HEAD Write-Log "" @@ -771,46 +1489,124 @@ foreach ($file in $RevertableFiles) { Write-Log " ✓ $($RevertableFiles.Count) fix file(s) restored from HEAD" -# Step 4: Run tests WITH fix +# Step 4: Run ALL tests WITH fix +Write-Host "" +Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Cyan +Write-Host "║ STEP 4: Running tests WITH fix (expect PASS) ║" -ForegroundColor Cyan +Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Cyan Write-Log "" -Write-Log "==========================================" Write-Log "STEP 4: Running tests WITH fix (should PASS)" -Write-Log "==========================================" -& $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $WithFixLog +$withFixResults = @() +$testIndex = 0 +foreach ($testEntry in $AllDetectedTests) { + $testIndex++ + $icon = switch ($testEntry.Type) { "UITest" { "🖥️" } "DeviceTest" { "📱" } "UnitTest" { "🧪" } "XamlUnitTest" { "📄" } default { "❓" } } + + $sanitizedName = ($testEntry.TestName -replace '[^a-zA-Z0-9_\-\.]', '_') + if ($sanitizedName.Length -gt 60) { $sanitizedName = $sanitizedName.Substring(0, 60) } + $testLogFile = Join-Path $OutputPath "test-with-fix-$sanitizedName.log" + + # AzDO collapsible group for raw test output + Write-Host "##[group]🟢 WITH FIX $testIndex/$($AllDetectedTests.Count): $icon $($testEntry.TestName) (filter: $($testEntry.Filter))" + + $sw = [System.Diagnostics.Stopwatch]::StartNew() + try { + $result = Invoke-TestRunWithRetry -TestEntry $testEntry -LogFile $testLogFile + } catch { + $result = @{ Passed = $false; Failed = 0; Total = 0; PassCount = 0; FailCount = 0; Skipped = 0; EnvError = $true; Error = $_.Exception.Message } + Write-Host " ⚠️ Test invocation threw: $($_.Exception.Message)" -ForegroundColor Yellow + } + $sw.Stop() + $result.TestName = $testEntry.TestName + $result.TestType = $testEntry.Type + $result.Duration = $sw.Elapsed + $withFixResults += $result + + # Print raw log inside the collapsible group + if (Test-Path $testLogFile) { + $logLines = Get-Content $testLogFile -ErrorAction SilentlyContinue + $lineCount = if ($logLines) { $logLines.Count } else { 0 } + Write-Host " ── Log ($lineCount lines) ──" -ForegroundColor DarkGray + if ($logLines) { $logLines | ForEach-Object { Write-Host " $_" } } + } + Write-Host "##[endgroup]" + + # Print result OUTSIDE the group so it's always visible + $durStr = "$([math]::Round($sw.Elapsed.TotalSeconds))s" + $counts = if ($result.Total -gt 0) { " ($($result.Total) total, $($result.Failed) failed)" } else { "" } + if ($result.EnvError) { + Write-Host " ⚠️ $($testEntry.TestName): ENV ERROR$counts — $durStr — $($result.Error)" -ForegroundColor Yellow + } elseif ($result.Passed) { + Write-Host " ✅ $($testEntry.TestName): PASSED$counts — $durStr (expected)" -ForegroundColor Green + } else { + Write-Host " ❌ $($testEntry.TestName): FAILED$counts — $durStr (unexpected!)" -ForegroundColor Red + if ($result.FailureReason) { Write-Host " └─ $($result.FailureReason)" -ForegroundColor DarkGray } + } + Write-Log " [$($testEntry.Type)] $($testEntry.TestName): Passed=$($result.Passed) Failed=$($result.Failed) [$durStr]" +} + +# Combine into a single summary for backward compatibility +$withFixResult = @{ + Passed = ($withFixResults | Where-Object { -not $_.Passed }).Count -eq 0 + PassCount = ($withFixResults | Measure-Object -Property PassCount -Sum).Sum + FailCount = ($withFixResults | Measure-Object -Property FailCount -Sum).Sum + Failed = ($withFixResults | Measure-Object -Property Failed -Sum).Sum + Skipped = ($withFixResults | Measure-Object -Property Skipped -Sum).Sum + Total = ($withFixResults | Measure-Object -Property Total -Sum).Sum +} -$withFixResult = Get-TestResultFromLog -LogFile (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log") +$withFixResults | ForEach-Object { "[$($_.TestType)] $($_.TestName): Passed=$($_.Passed) Failed=$($_.Failed)" } | Out-File $WithFixLog -Append # Step 5: Evaluate results +Write-Host "" +Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor White +Write-Host "║ GATE SUMMARY ║" -ForegroundColor White +Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor White Write-Log "" -Write-Log "==========================================" Write-Log "VERIFICATION RESULTS" -Write-Log "==========================================" $verificationPassed = $false -$failedWithoutFix = -not $withoutFixResult.Passed -$passedWithFix = $withFixResult.Passed +# "Without fix" should FAIL → all tests should NOT pass +$failedWithoutFix = ($withoutFixResults | Where-Object { $_.Passed }).Count -eq 0 +# "With fix" should PASS → all tests should pass +$passedWithFix = ($withFixResults | Where-Object { -not $_.Passed }).Count -eq 0 -if ($failedWithoutFix) { - Write-Log "✅ Tests FAILED without fix (expected - issue detected)" -} else { - Write-Log "❌ Tests PASSED without fix (unexpected!)" - Write-Log " The tests don't detect the issue." -} +# Print a clear comparison table +Write-Host "" +Write-Host " Test Name │ Without Fix │ With Fix " -ForegroundColor White +Write-Host " ───────────────────────┼─────────────┼────────────" -ForegroundColor DarkGray +foreach ($t in $AllDetectedTests) { + $woResult = $withoutFixResults | Where-Object { $_.TestName -eq $t.TestName } + $wResult = $withFixResults | Where-Object { $_.TestName -eq $t.TestName } -if ($passedWithFix) { - Write-Log "✅ Tests PASSED with fix (expected - fix works)" -} else { - Write-Log "❌ Tests FAILED with fix (unexpected!)" - Write-Log " The fix doesn't resolve the issue, or there's another problem." + $woIcon = if ($woResult.EnvError) { "⚠️ ENV ERR" } elseif (-not $woResult.Passed) { "✅ FAIL " } else { "❌ PASS " } + $wIcon = if ($wResult.EnvError) { "⚠️ ENV ERR" } elseif ($wResult.Passed) { "✅ PASS " } else { "❌ FAIL " } + + $nameDisplay = $t.TestName + if ($nameDisplay.Length -gt 22) { $nameDisplay = $nameDisplay.Substring(0, 19) + "..." } + $nameDisplay = $nameDisplay.PadRight(22) + + $woColor = if ($woResult.EnvError) { "Yellow" } elseif (-not $woResult.Passed) { "Green" } else { "Red" } + $wColor = if ($wResult.EnvError) { "Yellow" } elseif ($wResult.Passed) { "Green" } else { "Red" } + + Write-Host " $nameDisplay │ " -NoNewline -ForegroundColor White + Write-Host "$woIcon" -NoNewline -ForegroundColor $woColor + Write-Host " │ " -NoNewline -ForegroundColor White + Write-Host "$wIcon" -ForegroundColor $wColor + + Write-Log " [$($t.Type)] $($t.TestName): without fix=$(if (-not $woResult.Passed) {'FAIL ✅'} else {'PASS ❌'}), with fix=$(if ($wResult.Passed) {'PASS ✅'} else {'FAIL ❌'})" } +Write-Host " ───────────────────────┼─────────────┼────────────" -ForegroundColor DarkGray +Write-Host " Expected │ FAIL │ PASS " -ForegroundColor DarkGray +Write-Host "" $verificationPassed = $failedWithoutFix -and $passedWithFix Write-Log "" Write-Log "Summary:" -Write-Log " - Tests WITHOUT fix: $(if ($failedWithoutFix) { 'FAIL ✅ (expected)' } else { 'PASS ❌ (should fail!)' })" -Write-Log " - Tests WITH fix: $(if ($passedWithFix) { 'PASS ✅ (expected)' } else { 'FAIL ❌ (should pass!)' })" +Write-Log " - Tests WITHOUT fix: $(if ($failedWithoutFix) { 'ALL FAIL ✅ (expected)' } else { 'SOME PASS ❌ (should all fail!)' })" +Write-Log " - Tests WITH fix: $(if ($passedWithFix) { 'ALL PASS ✅ (expected)' } else { 'SOME FAIL ❌ (should all pass!)' })" # Generate markdown report Write-MarkdownReport ` @@ -818,7 +1614,15 @@ Write-MarkdownReport ` -FailedWithoutFix $failedWithoutFix ` -PassedWithFix $passedWithFix ` -WithoutFixResult $withoutFixResult ` - -WithFixResult $withFixResult + -WithFixResult $withFixResult ` + -WithoutFixResultsList $withoutFixResults ` + -WithFixResultsList $withFixResults ` + -Tests $AllDetectedTests ` + -ReportMergeBase $MergeBase ` + -ReportPlatform $Platform ` + -ReportBaseBranch $BaseBranchName ` + -ReportRevertableFiles $RevertableFiles ` + -ReportNewFiles $NewFiles if ($verificationPassed) { Write-Host "" diff --git a/eng/pipelines/ci-copilot.yml b/eng/pipelines/ci-copilot.yml index a6777956a5db..655d4423f8a7 100644 --- a/eng/pipelines/ci-copilot.yml +++ b/eng/pipelines/ci-copilot.yml @@ -275,22 +275,30 @@ stages: # Prepare emulator for CI use — keeps device responsive during idle period echo "=== Preparing emulator for CI ===" # Wait for device to stabilize after boot (transient offline state) - for i in $(seq 1 10); do + DEVICE_READY=false + for i in $(seq 1 20); do if adb -s $DEVICE_ID shell echo ok 2>/dev/null | grep -q ok; then + DEVICE_READY=true break fi - echo "Device offline, retrying ($i/10)..." - sleep 3 + echo "Device offline, retrying ($i/20)..." + sleep 5 done + + if [ "$DEVICE_READY" = false ]; then + echo "##vso[task.logissue type=error]Emulator went offline after boot — device not responsive after 100s" + exit 1 + fi + # Disable all animations (reduces CPU load and flakiness) - adb -s $DEVICE_ID shell settings put global window_animation_scale 0.0 - adb -s $DEVICE_ID shell settings put global transition_animation_scale 0.0 - adb -s $DEVICE_ID shell settings put global animator_duration_scale 0.0 + adb -s $DEVICE_ID shell settings put global window_animation_scale 0.0 || true + adb -s $DEVICE_ID shell settings put global transition_animation_scale 0.0 || true + adb -s $DEVICE_ID shell settings put global animator_duration_scale 0.0 || true # Prevent screen from turning off (emulator simulates AC charging) - adb -s $DEVICE_ID shell settings put system screen_off_timeout 2147483647 - adb -s $DEVICE_ID shell svc power stayon true + adb -s $DEVICE_ID shell settings put system screen_off_timeout 2147483647 || true + adb -s $DEVICE_ID shell svc power stayon true || true # Wake screen and dismiss any lock screen - adb -s $DEVICE_ID shell input keyevent 82 + adb -s $DEVICE_ID shell input keyevent 82 || true sleep 1 # Dismiss any "System UI has stopped" or crash dialogs adb -s $DEVICE_ID shell am broadcast -a android.intent.action.CLOSE_SYSTEM_DIALOGS 2>/dev/null || true @@ -302,7 +310,7 @@ stages: echo "##vso[task.prependpath]$ANDROID_SDK_ROOT/platform-tools" echo "##vso[task.prependpath]$ANDROID_SDK_ROOT/emulator" displayName: 'Create AVD and Boot Android Emulator' - retryCountOnTaskFailure: 1 + retryCountOnTaskFailure: 3 timeoutInMinutes: 15 # Install Node.js and Appium (same as ui-tests-steps.yml)