Improve test report formatting and summary extraction#33793
Improve test report formatting and summary extraction#33793PureWeen merged 2 commits intodotnet:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refines the formatting of test verification reports to make them more suitable for embedding in PR comments. The changes focus on making the reports less prominent by downgrading heading levels and improving the presentation of verification results.
Changes:
- Downgrade report heading levels in verify-tests-fail.ps1 (H1→H3, H2→H4) for a more embeddable format
- Extract concise summaries by trimming content after "Test Results Details" section
- Simplify status presentation with lowercase labels ("passed"/"failed") and collapsible verification blocks
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 | Downgrades main section headings from H1/H2 to H3/H4 to reduce visual prominence in embedded reports |
| .github/skills/ai-summary-comment/scripts/post-verify-tests-comment.ps1 | Adds regex-based summary extraction, changes status labels to lowercase, and wraps verification in collapsible details with status in summary tag |
Comments suppressed due to low confidence (1)
.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1:660
- The heading hierarchy is inverted in the updated report structure. The main section "Test Results Details" is now H4 (####), but its subsections "Test Run 1: WITHOUT Fix" and "Test Run 2: WITH Fix" remain at H3 (###). This creates an incorrect hierarchy where H3 subsections appear under an H4 parent.
For proper heading structure, these subsections should be downgraded to H5 (#####) to maintain the hierarchy:
- H3: Test Verification Report
- H4: Test Results Details
- H5: Test Run 1: WITHOUT Fix
- H5: Test Run 2: WITH Fix
The same issue affects "Fix Files" and "New Files (Not Reverted)" subsections under the H4 "Configuration" section - they should also be H5.
### Test Run 1: WITHOUT Fix
**Expected:** Tests should FAIL (bug is present)
**Actual:** Tests $(if ($FailedWithoutFix) { "FAILED" } else { "PASSED" }) $(if ($FailedWithoutFix) { "✅" } else { "❌" })
**Test Summary:**
- Total: $($WithoutFixResult.Total)
- Passed: $($WithoutFixResult.Passed)
- Failed: $($WithoutFixResult.Failed)
- Skipped: $($WithoutFixResult.Skipped)
$(if ($WithoutFixResult.FailureReason) {
"**Failure Reason:** ``$($WithoutFixResult.FailureReason)``"
})
<details>
<summary>View full test output (without fix)</summary>
$(Get-Content $WithoutFixLog -Raw)
</details>
---
### Test Run 2: WITH Fix
```
</details>
.github/skills/ai-summary-comment/scripts/post-verify-tests-comment.ps1
Outdated
Show resolved
Hide resolved
Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible <details> with the status shown in the <summary>. In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable.
ff997b2 to
57080cb
Compare
📋 PR Finalization ReviewTitle: ✅ GoodCurrent: Description: ✅ GoodDescription needs updates. See details below. **
Recommended Description:
Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible with the status shown in the . In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable. |
PowerShell's \ is a read-only automatic variable that contains recent errors. Using it in a foreach loop causes 'Cannot overwrite variable Error' error. Renamed to \ in two locations: - Line 402: foreach loop iterating result.Errors - Line 438: foreach loop iterating allValidationErrors
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible <details> with the status shown in the <summary>. In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable. --------- Co-authored-by: Shane Neuville <shane94@hotmail.com>
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible
. In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable.