diff --git a/.github/agents/pr.md b/.github/agents/pr.md index b2c6fbe67805..c7d4d8600e12 100644 --- a/.github/agents/pr.md +++ b/.github/agents/pr.md @@ -407,7 +407,7 @@ Tests were already verified to FAIL in Phase 2. Gate is a confirmation step: Use full verification mode - tests should FAIL without fix, PASS with fix. ```bash -pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android +pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification ``` ### Expected Output (PR with fix) diff --git a/.github/skills/verify-tests-fail-without-fix/SKILL.md b/.github/skills/verify-tests-fail-without-fix/SKILL.md index f0708157e4c5..37f9a649cf0e 100644 --- a/.github/skills/verify-tests-fail-without-fix/SKILL.md +++ b/.github/skills/verify-tests-fail-without-fix/SKILL.md @@ -1,89 +1,116 @@ --- name: verify-tests-fail-without-fix -description: Verifies UI tests catch the bug. Auto-detects mode based on git diff - if fix files exist, verifies FAIL without fix and PASS with fix. If only test files, verifies tests FAIL. +description: Verifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation). --- # Verify Tests Fail Without Fix -Verifies UI tests actually catch the issue. **Mode is auto-detected based on git diff.** +Verifies UI tests actually catch the issue. Supports two workflow modes: -## Usage +## Mode 1: Verify Failure Only (Test Creation) + +Use when **creating tests before writing a fix**: +- Runs tests to verify they **FAIL** (proving they catch the bug) +- No fix files required +- Perfect for test-first development ```bash -# Auto-detects everything - just specify platform +# Auto-detect test filter from changed test files pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android # With explicit test filter pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" ``` -## Auto-Detection +## Mode 2: Full Verification (Fix Validation) + +Use when **validating both tests and fix**: +1. **Without fix** - tests should FAIL (bug is present) +2. **With fix** - tests should PASS (bug is fixed) + +```bash +# Auto-detect everything (recommended) +pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification -The script automatically determines the mode: +# With explicit test filter +pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification +``` -| Changed Files | Mode | Behavior | -|---------------|------|----------| -| Fix files + test files | Full verification | FAIL without fix, PASS with fix | -| Only test files | Verify failure only | Tests must FAIL (reproduce bug) | +**Note:** `-RequireFullVerification` ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode. -**Fix files** = any changed file NOT in test directories -**Test files** = files in `TestCases.*` directories +## Requirements + +**Verify Failure Only Mode:** +- Test files in the PR (or working directory) + +**Full Verification Mode:** +- Test files in the PR +- Fix files in the PR (non-test code changes) + +The script auto-detects which mode to use based on whether fix files are present. ## Expected Output -**Full mode (fix files detected):** +**Verify Failure Only Mode:** ``` ╔═══════════════════════════════════════════════════════════╗ ║ VERIFICATION PASSED ✅ ║ ╠═══════════════════════════════════════════════════════════╣ -║ - FAIL without fix (as expected) ║ -║ - PASS with fix (as expected) ║ +║ Tests FAILED as expected! ║ +║ This proves the tests correctly reproduce the bug. ║ ╚═══════════════════════════════════════════════════════════╝ ``` -**Verify failure only (no fix files):** +**Full Verification Mode:** ``` ╔═══════════════════════════════════════════════════════════╗ -║ VERIFICATION PASSED ✅ ║ +║ VERIFICATION PASSED ✅ ║ ╠═══════════════════════════════════════════════════════════╣ -║ Tests FAILED as expected (bug is reproduced) ║ +║ - FAIL without fix (as expected) ║ +║ - PASS with fix (as expected) ║ ╚═══════════════════════════════════════════════════════════╝ ``` +## What It Does + +**Verify Failure Only Mode (no fix files):** +1. Fetches base branch from origin (if available) +2. Auto-detects test classes from changed test files +3. Runs tests (should FAIL to prove they catch the bug) +4. Reports result + +**Full Verification Mode (fix files detected):** +1. Fetches base branch from origin to ensure accurate diff +2. Auto-detects fix files (non-test code) from git diff +3. Auto-detects test classes from `TestCases.Shared.Tests/*.cs` +4. Reverts fix files to base branch +5. Runs tests (should FAIL without fix) +6. Restores fix files +7. Runs tests (should PASS with fix) +8. Reports result + ## Troubleshooting | Problem | Cause | Solution | |---------|-------|----------| +| No fix files detected | Base branch detection failed or no non-test files changed | Use `-FixFiles` or `-BaseBranch` explicitly | | Tests pass without fix | Tests don't detect the bug | Review test assertions, update test | -| Tests pass (no fix files) | **Test is wrong** | Review test vs issue description, fix test | +| Tests fail with fix | Fix doesn't work or test is wrong | Review fix implementation | | App crashes | Duplicate issue numbers, XAML error | Check device logs | | Element not found | Wrong AutomationId, app crashed | Verify IDs match | -## What It Does - -**Full mode:** -1. Auto-detects fix files (non-test code) from git diff -2. Auto-detects test classes from `TestCases.Shared.Tests/*.cs` -3. Reverts fix files to base branch -4. Runs tests (should FAIL without fix) -5. Restores fix files -6. Runs tests (should PASS with fix) -7. Reports result - -**Verify Failure Only mode:** -1. Runs tests once -2. Verifies they FAIL (bug reproduced) -3. Reports result - ## Optional Parameters ```bash +# Require full verification (fail if no fix files detected) - recommended +-RequireFullVerification + # Explicit test filter -TestFilter "Issue32030|ButtonUITests" # Explicit fix files -FixFiles @("src/Core/src/File.cs") -# Verify failure only (no fix exists yet) --VerifyFailureOnly +# Explicit base branch +-BaseBranch "main" ``` diff --git a/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 b/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 index 5aa8b570ab8e..8250d5314b86 100644 --- a/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 +++ b/.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 @@ -1,22 +1,23 @@ #!/usr/bin/env pwsh <# .SYNOPSIS - Verifies that UI tests catch the bug. Auto-detects mode based on whether fix files exist. + Verifies that UI tests catch the bug. Supports two modes: verify failure only or full verification. .DESCRIPTION - This script verifies that tests actually catch the issue. It auto-detects the mode: + This script verifies that tests actually catch the issue. It supports two modes: - **If fix files exist (non-test code changed):** - - Full verification mode - - Reverts fix files to base branch - - Runs tests WITHOUT fix (should FAIL) - - Restores fix files - - Runs tests WITH fix (should PASS) + VERIFY FAILURE ONLY MODE (no fix files detected): + - Runs tests to verify they FAIL (proving they catch the bug) + - Used when creating tests before writing a fix - **If only test files changed (no fix files):** - - Verify failure only mode - - Runs tests once expecting them to FAIL - - Confirms tests reproduce the bug + FULL VERIFICATION MODE (fix files detected): + 1. Reverting fix files to base branch + 2. Running tests WITHOUT fix (should FAIL) + 3. Restoring fix files + 4. Running tests WITH fix (should PASS) + + The script auto-detects which mode to use based on whether fix files are present. + Fix files and test filters are auto-detected from the git diff (non-test files that changed). .PARAMETER Platform Target platform: "android" or "ios" @@ -27,7 +28,7 @@ .PARAMETER FixFiles (Optional) Array of file paths to revert. If not provided, auto-detects from git diff - by excluding test directories. + by excluding test directories. If no fix files are found, runs in verify failure only mode. .PARAMETER BaseBranch Branch to revert files from. Auto-detected from PR if not specified. @@ -35,12 +36,21 @@ .PARAMETER OutputDir Directory to store results (default: "CustomAgentLogsTmp/TestValidation") +.PARAMETER RequireFullVerification + If set, the script will fail if it cannot run full verification mode + (i.e., if no fix files are detected). Without this flag, the script will + automatically run in verify failure only mode when no fix files are found. + .EXAMPLE - # Auto-detect everything - simplest usage + # Verify failure only mode - tests should fail (test creation workflow) ./verify-tests-fail.ps1 -Platform android .EXAMPLE - # Specify test filter, auto-detect mode and fix files + # Full verification mode - require fix files to be present + ./verify-tests-fail.ps1 -Platform android -RequireFullVerification + +.EXAMPLE + # Specify test filter explicitly (works in both modes) ./verify-tests-fail.ps1 -Platform android -TestFilter "Issue32030" .EXAMPLE @@ -64,7 +74,10 @@ param( [string]$BaseBranch, [Parameter(Mandatory = $false)] - [string]$OutputDir = "CustomAgentLogsTmp/TestValidation" + [string]$OutputDir = "CustomAgentLogsTmp/TestValidation", + + [Parameter(Mandatory = $false)] + [switch]$RequireFullVerification ) $ErrorActionPreference = "Stop" @@ -88,7 +101,7 @@ $TestPathPatterns = @( # Function to check if a file should be excluded from fix files function Test-IsTestFile { param([string]$FilePath) - + foreach ($pattern in $TestPathPatterns) { if ($FilePath -like $pattern) { return $true @@ -98,116 +111,390 @@ function Test-IsTestFile { } # ============================================================ -# AUTO-DETECT MODE: Check if there are fix files to revert +# Find the merge-base commit (where current branch diverged from base) +# This is more robust than tracking branch names/refs +# For fork workflows: fetches directly from the PR's target repo URL +# so it works even if the fork's main branch is out of sync # ============================================================ +function Find-MergeBase { + param([string]$ExplicitBaseBranch) + + # 1. If explicit base branch provided, use it directly + if ($ExplicitBaseBranch) { + # Try with origin/ prefix first, then without + foreach ($ref in @("origin/$ExplicitBaseBranch", $ExplicitBaseBranch)) { + $mergeBase = git merge-base HEAD $ref 2>$null + if ($mergeBase) { + return @{ MergeBase = $mergeBase; BaseBranch = $ExplicitBaseBranch; Source = "explicit" } + } + } + } -# Try to detect base branch -$BaseBranchDetected = $BaseBranch -if (-not $BaseBranchDetected) { - $currentBranch = git rev-parse --abbrev-ref HEAD 2>$null - $remote = git config "branch.$currentBranch.remote" 2>$null - if (-not $remote) { $remote = "origin" } - - $remoteUrl = git remote get-url $remote 2>$null - $repo = $null - if ($remoteUrl -match "github\.com[:/]([^/]+/[^/]+?)(\.git)?$") { - $repo = $matches[1] + # 2. Try to get PR metadata including the TARGET repository + # This is critical for fork workflows where origin points to the fork, + # not the upstream repo. We fetch directly from the target repo URL. + # The PR URL contains the target repo: https://github.com/OWNER/REPO/pull/123 + $prJson = gh pr view --json baseRefName,url 2>$null + if ($prJson) { + $prInfo = $prJson | ConvertFrom-Json + $prBaseBranch = $prInfo.baseRefName + $prUrl = $prInfo.url + + # Parse owner/repo from PR URL: https://github.com/OWNER/REPO/pull/123 + $targetOwner = $null + $targetRepo = $null + if ($prUrl -match "github\.com/([^/]+)/([^/]+)/pull/") { + $targetOwner = $matches[1] + $targetRepo = $matches[2] + } + + if ($prBaseBranch -and $targetOwner -and $targetRepo) { + # Construct the target repo URL and fetch directly from it + # This works even if the developer hasn't set up an 'upstream' remote + # and even if their fork's main is completely out of sync + $targetUrl = "https://github.com/$targetOwner/$targetRepo.git" + Write-Host "ℹ️ PR targets $targetOwner/$targetRepo - fetching $prBaseBranch from upstream..." -ForegroundColor Cyan + git fetch $targetUrl $prBaseBranch 2>$null + + if ($LASTEXITCODE -eq 0) { + # FETCH_HEAD now points to the target repo's base branch + $mergeBase = git merge-base HEAD FETCH_HEAD 2>$null + if ($mergeBase) { + return @{ MergeBase = $mergeBase; BaseBranch = $prBaseBranch; Source = "pr-target-repo"; TargetRepo = "$targetOwner/$targetRepo" } + } + } + } + + # Fallback: try fetching from origin (works if origin IS the target repo) + if ($prBaseBranch) { + git fetch origin $prBaseBranch 2>$null + foreach ($ref in @("origin/$prBaseBranch", $prBaseBranch)) { + $mergeBase = git merge-base HEAD $ref 2>$null + if ($mergeBase) { + return @{ MergeBase = $mergeBase; BaseBranch = $prBaseBranch; Source = "pr-metadata" } + } + } + } } - - if ($repo) { - $BaseBranchDetected = gh pr view $currentBranch --repo $repo --json baseRefName --jq '.baseRefName' 2>$null - } else { - $BaseBranchDetected = gh pr view --json baseRefName --jq '.baseRefName' 2>$null + + # 3. Fallback: Find closest merge-base among common base branch patterns + # The "correct" base is the one with fewest commits between merge-base and HEAD + Write-Host "ℹ️ No PR detected, scanning remote branches for closest base..." -ForegroundColor Cyan + + # Fetch all remote refs to ensure we have latest + git fetch origin 2>$null + + # Get remote branches matching common base branch patterns + $remoteBranches = git branch -r --format='%(refname:short)' 2>$null | Where-Object { + $_ -match '^origin/(main|master|net\d+\.\d+|release/.*)$' } + + $bestMatch = $null + $shortestDistance = [int]::MaxValue + + foreach ($branch in $remoteBranches) { + $mergeBase = git merge-base HEAD $branch 2>$null + if ($mergeBase) { + $distance = [int](git rev-list --count "$mergeBase..HEAD" 2>$null) + if ($distance -lt $shortestDistance) { + $shortestDistance = $distance + $branchName = $branch -replace '^origin/', '' + $bestMatch = @{ MergeBase = $mergeBase; BaseBranch = $branchName; Source = "closest-merge-base"; Distance = $distance } + } + } + } + + return $bestMatch } -# Check for fix files (non-test files that changed) -$DetectedFixFiles = @() -if ($BaseBranchDetected) { - $changedFiles = git diff $BaseBranchDetected HEAD --name-only 2>$null - if ($LASTEXITCODE -ne 0) { - $changedFiles = git diff "origin/$BaseBranchDetected" HEAD --name-only 2>$null +# ============================================================ +# Auto-detect test filter from changed files +# ============================================================ +function Get-AutoDetectedTestFilter { + param([string]$MergeBase) + + $changedFiles = @() + if ($MergeBase) { + $changedFiles = git diff $MergeBase HEAD --name-only 2>$null } - - if ($changedFiles) { - foreach ($file in $changedFiles) { - if (-not (Test-IsTestFile $file)) { - $DetectedFixFiles += $file + + # If no merge-base, use git status to find unstaged/staged files + if (-not $changedFiles -or $changedFiles.Count -eq 0) { + $changedFiles = git diff --name-only 2>$null + if (-not $changedFiles -or $changedFiles.Count -eq 0) { + $changedFiles = git diff --cached --name-only 2>$null + } + } + + # Find test files (files in test directories that are .cs files) + $testFiles = @() + foreach ($file in $changedFiles) { + if ($file -match "TestCases\.(Shared\.Tests|HostApp).*\.cs$" -and $file -notmatch "^_") { + $testFiles += $file + } + } + + if ($testFiles.Count -eq 0) { + return $null + } + + # Extract class names from test files + $testClassNames = @() + foreach ($file in $testFiles) { + if ($file -match "TestCases\.Shared\.Tests.*\.cs$") { + $fullPath = Join-Path $RepoRoot $file + if (Test-Path $fullPath) { + $content = Get-Content $fullPath -Raw + if ($content -match "public\s+(partial\s+)?class\s+(\w+)") { + $className = $matches[2] + if ($className -notmatch "^_" -and $testClassNames -notcontains $className) { + $testClassNames += $className + } + } } } } + + # Fallback: use file names without extension + if ($testClassNames.Count -eq 0) { + foreach ($file in $testFiles) { + $fileName = [System.IO.Path]::GetFileNameWithoutExtension($file) + if ($fileName -notmatch "^_" -and $testClassNames -notcontains $fileName) { + $testClassNames += $fileName + } + } + } + + if ($testClassNames.Count -eq 0) { + return $null + } + + return @{ + Filter = if ($testClassNames.Count -eq 1) { $testClassNames[0] } else { $testClassNames -join "|" } + ClassNames = $testClassNames + } +} + +# ============================================================ +# Parse test results from log file +# ============================================================ +function Get-TestResultFromLog { + param([string]$LogFile) + + if (-not (Test-Path $LogFile)) { + return @{ Passed = $false; Error = "Test output log not found: $LogFile" } + } + + $content = Get-Content $LogFile -Raw + + # Check for failures first - but only if count > 0 + if ($content -match "Failed:\s*(\d+)") { + $failCount = [int]$matches[1] + if ($failCount -gt 0) { + return @{ Passed = $false; FailCount = $failCount } + } + } + + # Check for passes + if ($content -match "Passed:\s*(\d+)") { + $passCount = [int]$matches[1] + if ($passCount -gt 0) { + return @{ Passed = $true; PassCount = $passCount } + } + } + + return @{ Passed = $false; Error = "Could not parse test results" } +} + +# ============================================================ +# AUTO-DETECT MODE: Find merge-base and fix files +# ============================================================ + +Write-Host "" +Write-Host "🔍 Detecting base branch and merge point..." -ForegroundColor Cyan + +$baseInfo = Find-MergeBase -ExplicitBaseBranch $BaseBranch + +if (-not $baseInfo) { + Write-Host "" + Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red + Write-Host "║ ERROR: COULD NOT FIND MERGE BASE ║" -ForegroundColor Red + Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Red + Write-Host "║ Could not determine where this branch diverged from. ║" -ForegroundColor Red + Write-Host "║ ║" -ForegroundColor Red + Write-Host "║ Tried: ║" -ForegroundColor Red + Write-Host "║ - PR metadata (gh pr view) ║" -ForegroundColor Red + Write-Host "║ - Common base branches (main, net*.0, release/*) ║" -ForegroundColor Red + Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red + Write-Host "" + Write-Host "To fix, specify -BaseBranch explicitly:" -ForegroundColor Cyan + Write-Host " ./verify-tests-fail.ps1 -Platform android -BaseBranch main" -ForegroundColor White + exit 1 +} + +$MergeBase = $baseInfo.MergeBase +$BaseBranchName = $baseInfo.BaseBranch + +if ($baseInfo.TargetRepo) { + Write-Host "✅ PR target: $($baseInfo.TargetRepo) ($BaseBranchName branch)" -ForegroundColor Green +} else { + Write-Host "✅ Base branch: $BaseBranchName (via $($baseInfo.Source))" -ForegroundColor Green +} +Write-Host "✅ Merge base commit: $($MergeBase.Substring(0, 8))" -ForegroundColor Green +if ($baseInfo.Distance) { + Write-Host " ($($baseInfo.Distance) commits ahead of $BaseBranchName)" -ForegroundColor Gray +} + +# Check for fix files (non-test files that changed since merge-base) +$DetectedFixFiles = @() +$changedFiles = git diff $MergeBase HEAD --name-only 2>$null + +if ($changedFiles) { + foreach ($file in $changedFiles) { + if (-not (Test-IsTestFile $file)) { + $DetectedFixFiles += $file + } + } } -# Also check explicitly provided fix files +# Override with explicitly provided fix files if ($FixFiles -and $FixFiles.Count -gt 0) { $DetectedFixFiles = $FixFiles } -# Determine mode based on whether we have fix files -$VerifyFailureOnlyMode = ($DetectedFixFiles.Count -eq 0) +# Error if no fix files detected and RequireFullVerification is set +if ($DetectedFixFiles.Count -eq 0 -and $RequireFullVerification) { + Write-Host "" + Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red + Write-Host "║ ERROR: NO FIX FILES DETECTED ║" -ForegroundColor Red + Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Red + Write-Host "║ Full verification mode required but no fix files found. ║" -ForegroundColor Red + Write-Host "║ ║" -ForegroundColor Red + Write-Host "║ Possible causes: ║" -ForegroundColor Red + Write-Host "║ - No non-test files changed since merge-base ║" -ForegroundColor Red + Write-Host "║ - All changes are in test directories ║" -ForegroundColor Red + Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red + Write-Host "" + Write-Host "Debug info:" -ForegroundColor Yellow + Write-Host " Merge base: $MergeBase" -ForegroundColor Yellow + Write-Host " Base branch: $BaseBranchName" -ForegroundColor Yellow + Write-Host " Current branch: $(git rev-parse --abbrev-ref HEAD)" -ForegroundColor Yellow + Write-Host "" + Write-Host "To fix, try one of:" -ForegroundColor Cyan + Write-Host " 1. Specify fix files explicitly: -FixFiles @('path/to/fix.cs')" -ForegroundColor White + Write-Host " 2. Remove -RequireFullVerification to run in failure-only mode" -ForegroundColor White + exit 1 +} -# ============================================================ -# VERIFY FAILURE ONLY MODE (no fix files detected) -# ============================================================ -if ($VerifyFailureOnlyMode) { +# If no fix files and not requiring full verification, run in "verify failure only" mode +if ($DetectedFixFiles.Count -eq 0) { Write-Host "" Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Cyan Write-Host "║ VERIFY FAILURE ONLY MODE ║" -ForegroundColor Cyan Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Cyan - Write-Host "║ No fix files detected - verifying tests FAIL ║" -ForegroundColor Cyan - Write-Host "║ (Only test files changed, or new tests created) ║" -ForegroundColor Cyan + Write-Host "║ No fix files detected - will only verify: ║" -ForegroundColor Cyan + Write-Host "║ 1. Tests FAIL (proving they catch the bug) ║" -ForegroundColor Cyan + Write-Host "║ ║" -ForegroundColor Cyan + Write-Host "║ Use this mode when creating tests before writing a fix. ║" -ForegroundColor Cyan Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Cyan Write-Host "" - + + # Auto-detect test filter if not provided if (-not $TestFilter) { - Write-Host "❌ -TestFilter is required when no fix files are detected" -ForegroundColor Red - Write-Host " Example: -TestFilter 'Issue33356'" -ForegroundColor Yellow - exit 1 + Write-Host "🔍 Auto-detecting test filter from changed test files..." -ForegroundColor Cyan + $filterResult = Get-AutoDetectedTestFilter -MergeBase $MergeBase + + if (-not $filterResult) { + Write-Host "❌ Could not auto-detect test filter. No test files found in changed files." -ForegroundColor Red + Write-Host " Looking for files matching: TestCases.(Shared.Tests|HostApp)/*.cs" -ForegroundColor Yellow + Write-Host " Please provide -TestFilter parameter explicitly." -ForegroundColor Yellow + exit 1 + } + + $TestFilter = $filterResult.Filter + Write-Host "✅ Auto-detected $($filterResult.ClassNames.Count) test class(es):" -ForegroundColor Green + foreach ($name in $filterResult.ClassNames) { + Write-Host " - $name" -ForegroundColor White + } + Write-Host " Filter: $TestFilter" -ForegroundColor Cyan } - + # Create output directory $OutputPath = Join-Path $RepoRoot $OutputDir New-Item -ItemType Directory -Force -Path $OutputPath | Out-Null - $FailureOnlyLog = Join-Path $OutputPath "verify-failure-only.log" - - Write-Host "Platform: $Platform" -ForegroundColor White - Write-Host "TestFilter: $TestFilter" -ForegroundColor White + + $ValidationLog = Join-Path $OutputPath "verification-log.txt" + $TestLog = Join-Path $OutputPath "test-failure-verification.log" + + # Initialize log + "" | Set-Content $ValidationLog + "=========================================" | Add-Content $ValidationLog + "Verify Tests Fail (Failure Only Mode)" | Add-Content $ValidationLog + "=========================================" | Add-Content $ValidationLog + "Platform: $Platform" | Add-Content $ValidationLog + "TestFilter: $TestFilter" | Add-Content $ValidationLog + "MergeBase: $MergeBase" | Add-Content $ValidationLog + "" | Add-Content $ValidationLog + + Write-Host "🧪 Running tests (expecting them to FAIL)..." -ForegroundColor Cyan Write-Host "" - Write-Host "Running tests (expecting FAILURE)..." -ForegroundColor Yellow - - # Run the test + + # Use shared BuildAndRunHostApp.ps1 infrastructure with -Rebuild to ensure clean builds $buildScript = Join-Path $RepoRoot ".github/scripts/BuildAndRunHostApp.ps1" - & $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $FailureOnlyLog - - # Check test result + & $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $TestLog + + # Parse test results using shared function $testOutputLog = Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log" - $testFailed = $false - - if (Test-Path $testOutputLog) { - $content = Get-Content $testOutputLog -Raw - if ($content -match "Failed:\s*(\d+)" -and [int]$matches[1] -gt 0) { - $testFailed = $true - } - } - + $testResult = Get-TestResultFromLog -LogFile $testOutputLog + + # Evaluate results + Write-Host "" + Write-Host "==========================================" + Write-Host "VERIFICATION RESULTS" + Write-Host "==========================================" Write-Host "" - if ($testFailed) { + + if ($testResult.Error) { + Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red + Write-Host "║ ERROR PARSING TEST RESULTS ║" -ForegroundColor Red + Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Red + Write-Host "║ $($testResult.Error.PadRight(57)) ║" -ForegroundColor Red + Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red + exit 1 + } + + if (-not $testResult.Passed) { + # Tests FAILED - this is what we want! Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Green - Write-Host "║ VERIFICATION PASSED ✅ ║" -ForegroundColor Green + Write-Host "║ VERIFICATION PASSED ✅ ║" -ForegroundColor Green Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Green - Write-Host "║ Tests FAILED as expected (bug is reproduced) ║" -ForegroundColor Green + Write-Host "║ Tests FAILED as expected! ║" -ForegroundColor Green Write-Host "║ ║" -ForegroundColor Green - Write-Host "║ Next: Implement a fix, then rerun to verify tests pass. ║" -ForegroundColor Green + Write-Host "║ This proves the tests correctly reproduce the bug. ║" -ForegroundColor Green + Write-Host "║ You can now proceed to write the fix. ║" -ForegroundColor Green Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Green + Write-Host "" + Write-Host "Failed tests: $($testResult.FailCount)" -ForegroundColor Yellow exit 0 } else { + # Tests PASSED - this is bad! Write-Host "╔═══════════════════════════════════════════════════════════╗" -ForegroundColor Red - Write-Host "║ VERIFICATION FAILED ❌ ║" -ForegroundColor Red + Write-Host "║ VERIFICATION FAILED ❌ ║" -ForegroundColor Red Write-Host "╠═══════════════════════════════════════════════════════════╣" -ForegroundColor Red - Write-Host "║ Tests PASSED (unexpected - bug not reproduced) ║" -ForegroundColor Red + Write-Host "║ Tests PASSED but they should FAIL! ║" -ForegroundColor Red Write-Host "║ ║" -ForegroundColor Red - Write-Host "║ Your test is wrong. Fix it and rerun. ║" -ForegroundColor Red + Write-Host "║ This means your tests don't actually reproduce the bug. ║" -ForegroundColor Red + Write-Host "║ ║" -ForegroundColor Red + Write-Host "║ Possible causes: ║" -ForegroundColor Red + Write-Host "║ 1. Test scenario doesn't match the issue description ║" -ForegroundColor Red + Write-Host "║ 2. Test assertions are wrong or too lenient ║" -ForegroundColor Red + Write-Host "║ 3. The bug was already fixed in this branch ║" -ForegroundColor Red + Write-Host "║ 4. The bug only happens in specific conditions ║" -ForegroundColor Red + Write-Host "║ ║" -ForegroundColor Red + Write-Host "║ Go back and revise your tests! ║" -ForegroundColor Red Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Red + Write-Host "" + Write-Host "Passed tests: $($testResult.PassCount)" -ForegroundColor Yellow exit 1 } } @@ -226,10 +513,8 @@ Write-Host "║ 2. Tests PASS with fix ║" - Write-Host "╚═══════════════════════════════════════════════════════════╝" -ForegroundColor Cyan Write-Host "" -$BaseBranch = $BaseBranchDetected $FixFiles = $DetectedFixFiles -Write-Host "✅ Base branch: $BaseBranch" -ForegroundColor Green Write-Host "✅ Fix files ($($FixFiles.Count)):" -ForegroundColor Green foreach ($file in $FixFiles) { Write-Host " - $file" -ForegroundColor White @@ -238,68 +523,18 @@ foreach ($file in $FixFiles) { # Auto-detect test filter from test files if not provided if (-not $TestFilter) { Write-Host "🔍 Auto-detecting test filter from changed test files..." -ForegroundColor Cyan - - $changedFiles = git diff $BaseBranch HEAD --name-only 2>$null - if ($LASTEXITCODE -ne 0) { - $changedFiles = git diff "origin/$BaseBranch" HEAD --name-only 2>$null - } - - # Find test files (files in test directories that are .cs files) - $testFiles = @() - foreach ($file in $changedFiles) { - if ($file -match "TestCases\.(Shared\.Tests|HostApp).*\.cs$" -and $file -notmatch "^_") { - $testFiles += $file - } - } - - if ($testFiles.Count -eq 0) { + $filterResult = Get-AutoDetectedTestFilter -MergeBase $MergeBase + + if (-not $filterResult) { Write-Host "❌ Could not auto-detect test filter. No test files found in changed files." -ForegroundColor Red Write-Host " Looking for files matching: TestCases.(Shared.Tests|HostApp)/*.cs" -ForegroundColor Yellow Write-Host " Please provide -TestFilter parameter explicitly." -ForegroundColor Yellow exit 1 } - - # Extract class names from test files - $testClassNames = @() - foreach ($file in $testFiles) { - if ($file -match "TestCases\.Shared\.Tests.*\.cs$") { - $fullPath = Join-Path $RepoRoot $file - if (Test-Path $fullPath) { - $content = Get-Content $fullPath -Raw - if ($content -match "public\s+(partial\s+)?class\s+(\w+)") { - $className = $matches[2] - if ($className -notmatch "^_" -and $testClassNames -notcontains $className) { - $testClassNames += $className - } - } - } - } - } - - # Fallback: use file names without extension - if ($testClassNames.Count -eq 0) { - foreach ($file in $testFiles) { - $fileName = [System.IO.Path]::GetFileNameWithoutExtension($file) - if ($fileName -notmatch "^_" -and $testClassNames -notcontains $fileName) { - $testClassNames += $fileName - } - } - } - - if ($testClassNames.Count -eq 0) { - Write-Host "❌ Could not extract test class names from changed files." -ForegroundColor Red - Write-Host " Please provide -TestFilter parameter explicitly." -ForegroundColor Yellow - exit 1 - } - - if ($testClassNames.Count -eq 1) { - $TestFilter = $testClassNames[0] - } else { - $TestFilter = $testClassNames -join "|" - } - - Write-Host "✅ Auto-detected $($testClassNames.Count) test class(es):" -ForegroundColor Green - foreach ($name in $testClassNames) { + + $TestFilter = $filterResult.Filter + Write-Host "✅ Auto-detected $($filterResult.ClassNames.Count) test class(es):" -ForegroundColor Green + foreach ($name in $filterResult.ClassNames) { Write-Host " - $name" -ForegroundColor White } Write-Host " Filter: $TestFilter" -ForegroundColor Cyan @@ -321,20 +556,7 @@ function Write-Log { Add-Content -Path $ValidationLog -Value $logLine } -function Get-TestResult { - param([string]$LogFile) - - if (Test-Path $LogFile) { - $content = Get-Content $LogFile -Raw - if ($content -match "Failed:\s*(\d+)") { - return @{ Passed = $false; FailCount = [int]$matches[1] } - } - if ($content -match "Passed:\s*(\d+)") { - return @{ Passed = $true; PassCount = [int]$matches[1] } - } - } - return @{ Passed = $false; Error = "Could not parse test results" } -} +# Reuse the Get-TestResultFromLog function defined earlier # Initialize log "" | Set-Content $ValidationLog @@ -344,7 +566,8 @@ Write-Log "==========================================" Write-Log "Platform: $Platform" Write-Log "TestFilter: $TestFilter" Write-Log "FixFiles: $($FixFiles -join ', ')" -Write-Log "BaseBranch: $BaseBranch" +Write-Log "BaseBranch: $BaseBranchName" +Write-Log "MergeBase: $MergeBase" Write-Log "" # Verify fix files exist @@ -358,22 +581,19 @@ foreach ($file in $FixFiles) { Write-Log " ✓ $file exists" } -# Determine which files exist in the base branch (can be reverted) +# Determine which files exist at the merge-base (can be reverted) Write-Log "" -Write-Log "Checking which fix files exist in $BaseBranch..." +Write-Log "Checking which fix files exist at merge-base ($($MergeBase.Substring(0, 8)))..." $RevertableFiles = @() $NewFiles = @() foreach ($file in $FixFiles) { - # Check if file exists in base branch - $existsInBase = git ls-tree -r $BaseBranch --name-only -- $file 2>$null - if (-not $existsInBase) { - $existsInBase = git ls-tree -r "origin/$BaseBranch" --name-only -- $file 2>$null - } - + # Check if file exists at merge-base commit + $existsInBase = git ls-tree -r $MergeBase --name-only -- $file 2>$null + if ($existsInBase) { $RevertableFiles += $file - Write-Log " ✓ $file (exists in $BaseBranch - will revert)" + Write-Log " ✓ $file (exists at merge-base - will revert)" } else { $NewFiles += $file Write-Log " ○ $file (new file - skipping revert)" @@ -418,22 +638,22 @@ if ($uncommittedFiles.Count -gt 0) { Write-Log " ✓ All revertable fix files are committed" -# Step 1: Revert fix files to base branch +# Step 1: Revert fix files to merge-base state Write-Log "" Write-Log "==========================================" -Write-Log "STEP 1: Reverting fix files to $BaseBranch" +Write-Log "STEP 1: Reverting fix files to merge-base ($($MergeBase.Substring(0, 8)))" Write-Log "==========================================" foreach ($file in $RevertableFiles) { Write-Log " Reverting: $file" - git checkout $BaseBranch -- $file 2>&1 | Out-Null + git checkout $MergeBase -- $file 2>&1 | Out-Null if ($LASTEXITCODE -ne 0) { - Write-Log " Warning: Could not revert from $BaseBranch, trying origin/$BaseBranch" - git checkout "origin/$BaseBranch" -- $file 2>&1 | Out-Null + Write-Log " ERROR: Failed to revert $file from $MergeBase" + exit 1 } } -Write-Log " ✓ $($RevertableFiles.Count) fix file(s) reverted to $BaseBranch state" +Write-Log " ✓ $($RevertableFiles.Count) fix file(s) reverted to merge-base state" # Step 2: Run tests WITHOUT fix Write-Log "" @@ -445,7 +665,7 @@ Write-Log "==========================================" $buildScript = Join-Path $RepoRoot ".github/scripts/BuildAndRunHostApp.ps1" & $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $WithoutFixLog -$withoutFixResult = Get-TestResult -LogFile (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log") +$withoutFixResult = Get-TestResultFromLog -LogFile (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log") # Step 3: Restore fix files from current branch HEAD Write-Log "" @@ -472,7 +692,7 @@ Write-Log "==========================================" & $buildScript -Platform $Platform -TestFilter $TestFilter -Rebuild 2>&1 | Tee-Object -FilePath $WithFixLog -$withFixResult = Get-TestResult -LogFile (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log") +$withFixResult = Get-TestResultFromLog -LogFile (Join-Path $RepoRoot "CustomAgentLogsTmp/UITests/test-output.log") # Step 5: Evaluate results Write-Log ""