Skip to content
2 changes: 1 addition & 1 deletion .github/agents/pr.md
Original file line number Diff line number Diff line change
Expand Up @@ -407,7 +407,7 @@ Tests were already verified to FAIL in Phase 2. Gate is a confirmation step:
Use full verification mode - tests should FAIL without fix, PASS with fix.

```bash
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification
```

### Expected Output (PR with fix)
Expand Down
101 changes: 64 additions & 37 deletions .github/skills/verify-tests-fail-without-fix/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,89 +1,116 @@
---
name: verify-tests-fail-without-fix
description: Verifies UI tests catch the bug. Auto-detects mode based on git diff - if fix files exist, verifies FAIL without fix and PASS with fix. If only test files, verifies tests FAIL.
description: Verifies UI tests catch the bug. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
---

# Verify Tests Fail Without Fix

Verifies UI tests actually catch the issue. **Mode is auto-detected based on git diff.**
Verifies UI tests actually catch the issue. Supports two workflow modes:

## Usage
## Mode 1: Verify Failure Only (Test Creation)

Use when **creating tests before writing a fix**:
- Runs tests to verify they **FAIL** (proving they catch the bug)
- No fix files required
- Perfect for test-first development

```bash
# Auto-detects everything - just specify platform
# Auto-detect test filter from changed test files
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356"
```

## Auto-Detection
## Mode 2: Full Verification (Fix Validation)

Use when **validating both tests and fix**:
1. **Without fix** - tests should FAIL (bug is present)
2. **With fix** - tests should PASS (bug is fixed)

```bash
# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification

The script automatically determines the mode:
# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification
```

| Changed Files | Mode | Behavior |
|---------------|------|----------|
| Fix files + test files | Full verification | FAIL without fix, PASS with fix |
| Only test files | Verify failure only | Tests must FAIL (reproduce bug) |
**Note:** `-RequireFullVerification` ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.

**Fix files** = any changed file NOT in test directories
**Test files** = files in `TestCases.*` directories
## Requirements

**Verify Failure Only Mode:**
- Test files in the PR (or working directory)

**Full Verification Mode:**
- Test files in the PR
- Fix files in the PR (non-test code changes)

The script auto-detects which mode to use based on whether fix files are present.

## Expected Output

**Full mode (fix files detected):**
**Verify Failure Only Mode:**
```
╔═══════════════════════════════════════════════════════════╗
║ VERIFICATION PASSED ✅ ║
╠═══════════════════════════════════════════════════════════╣
- FAIL without fix (as expected)
- PASS with fix (as expected)
Tests FAILED as expected!
This proves the tests correctly reproduce the bug.
╚═══════════════════════════════════════════════════════════╝
```

**Verify failure only (no fix files):**
**Full Verification Mode:**
```
╔═══════════════════════════════════════════════════════════╗
VERIFICATION PASSED ✅
VERIFICATION PASSED ✅
╠═══════════════════════════════════════════════════════════╣
║ Tests FAILED as expected (bug is reproduced) ║
║ - FAIL without fix (as expected) ║
║ - PASS with fix (as expected) ║
╚═══════════════════════════════════════════════════════════╝
```

## What It Does

**Verify Failure Only Mode (no fix files):**
1. Fetches base branch from origin (if available)
2. Auto-detects test classes from changed test files
3. Runs tests (should FAIL to prove they catch the bug)
4. Reports result

**Full Verification Mode (fix files detected):**
1. Fetches base branch from origin to ensure accurate diff
2. Auto-detects fix files (non-test code) from git diff
3. Auto-detects test classes from `TestCases.Shared.Tests/*.cs`
4. Reverts fix files to base branch
5. Runs tests (should FAIL without fix)
6. Restores fix files
7. Runs tests (should PASS with fix)
8. Reports result

## Troubleshooting

| Problem | Cause | Solution |
|---------|-------|----------|
| No fix files detected | Base branch detection failed or no non-test files changed | Use `-FixFiles` or `-BaseBranch` explicitly |
| Tests pass without fix | Tests don't detect the bug | Review test assertions, update test |
| Tests pass (no fix files) | **Test is wrong** | Review test vs issue description, fix test |
| Tests fail with fix | Fix doesn't work or test is wrong | Review fix implementation |
| App crashes | Duplicate issue numbers, XAML error | Check device logs |
| Element not found | Wrong AutomationId, app crashed | Verify IDs match |

## What It Does

**Full mode:**
1. Auto-detects fix files (non-test code) from git diff
2. Auto-detects test classes from `TestCases.Shared.Tests/*.cs`
3. Reverts fix files to base branch
4. Runs tests (should FAIL without fix)
5. Restores fix files
6. Runs tests (should PASS with fix)
7. Reports result

**Verify Failure Only mode:**
1. Runs tests once
2. Verifies they FAIL (bug reproduced)
3. Reports result

## Optional Parameters

```bash
# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification

# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"

# Explicit fix files
-FixFiles @("src/Core/src/File.cs")

# Verify failure only (no fix exists yet)
-VerifyFailureOnly
# Explicit base branch
-BaseBranch "main"
```
Loading