Skip to content

Improve test report formatting and summary extraction#33793

Merged
PureWeen merged 2 commits intodotnet:mainfrom
kubaflo:post-verify-tests-comment-improvements
Jan 31, 2026
Merged

Improve test report formatting and summary extraction#33793
PureWeen merged 2 commits intodotnet:mainfrom
kubaflo:post-verify-tests-comment-improvements

Conversation

@kubaflo
Copy link
Contributor

@kubaflo kubaflo commented Jan 30, 2026

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible

with the status shown in the . In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable.

Copilot AI review requested due to automatic review settings January 30, 2026 13:39
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refines the formatting of test verification reports to make them more suitable for embedding in PR comments. The changes focus on making the reports less prominent by downgrading heading levels and improving the presentation of verification results.

Changes:

  • Downgrade report heading levels in verify-tests-fail.ps1 (H1→H3, H2→H4) for a more embeddable format
  • Extract concise summaries by trimming content after "Test Results Details" section
  • Simplify status presentation with lowercase labels ("passed"/"failed") and collapsible verification blocks

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 Downgrades main section headings from H1/H2 to H3/H4 to reduce visual prominence in embedded reports
.github/skills/ai-summary-comment/scripts/post-verify-tests-comment.ps1 Adds regex-based summary extraction, changes status labels to lowercase, and wraps verification in collapsible details with status in summary tag
Comments suppressed due to low confidence (1)

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1:660

  • The heading hierarchy is inverted in the updated report structure. The main section "Test Results Details" is now H4 (####), but its subsections "Test Run 1: WITHOUT Fix" and "Test Run 2: WITH Fix" remain at H3 (###). This creates an incorrect hierarchy where H3 subsections appear under an H4 parent.

For proper heading structure, these subsections should be downgraded to H5 (#####) to maintain the hierarchy:

  • H3: Test Verification Report
  • H4: Test Results Details
  • H5: Test Run 1: WITHOUT Fix
  • H5: Test Run 2: WITH Fix

The same issue affects "Fix Files" and "New Files (Not Reverted)" subsections under the H4 "Configuration" section - they should also be H5.

### Test Run 1: WITHOUT Fix

**Expected:** Tests should FAIL (bug is present)  
**Actual:** Tests $(if ($FailedWithoutFix) { "FAILED" } else { "PASSED" }) $(if ($FailedWithoutFix) { "✅" } else { "❌" })

**Test Summary:**
- Total: $($WithoutFixResult.Total)
- Passed: $($WithoutFixResult.Passed)
- Failed: $($WithoutFixResult.Failed)
- Skipped: $($WithoutFixResult.Skipped)

$(if ($WithoutFixResult.FailureReason) {
    "**Failure Reason:** ``$($WithoutFixResult.FailureReason)``"
})

<details>
<summary>View full test output (without fix)</summary>

$(Get-Content $WithoutFixLog -Raw)


</details>

---

### Test Run 2: WITH Fix
```
</details>

Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible <details> with the status shown in the <summary>. In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable.
@kubaflo kubaflo force-pushed the post-verify-tests-comment-improvements branch from ff997b2 to 57080cb Compare January 30, 2026 14:10
@rmarinho
Copy link
Member

📋 PR Finalization Review

Title: ✅ Good

Current: Improve test report formatting and summary extraction

Description: ✅ Good

Description needs updates. See details below.
Missing Elements:

**

  • NOTE block for testing artifacts

Recommended Description:

[!NOTE]
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Refine how test summaries are generated and adjust report markdown for embedding. In post-verify-tests-comment.ps1: extract a concise summary by trimming everything from the start up to the "## Test Results Details" section (regex fallback to full content if not found), lowercase the status labels to "passed/failed", and move the verification block into a collapsible

with the status shown in the . In verify-tests-fail.ps1: downgrade top-level headings to smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict, Configuration, Test Results Details, and Logs to produce a less-prominent, embeddable report. Changes are primarily formatting and presentation to keep PR comments concise and readable.

@dotnet dotnet deleted a comment from rmarinho Jan 30, 2026
@kubaflo kubaflo added the area-ai-agents Copilot CLI agents, agent skills, AI-assisted development label Jan 30, 2026
PowerShell's \ is a read-only automatic variable that contains recent errors.
Using it in a foreach loop causes 'Cannot overwrite variable Error' error.

Renamed to \ in two locations:
- Line 402: foreach loop iterating result.Errors
- Line 438: foreach loop iterating allValidationErrors
@PureWeen PureWeen merged commit 300f3f1 into dotnet:main Jan 31, 2026
2 of 3 checks passed
kubaflo added a commit to kubaflo/maui that referenced this pull request Feb 2, 2026
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

Refine how test summaries are generated and adjust report markdown for
embedding. In post-verify-tests-comment.ps1: extract a concise summary
by trimming everything from the start up to the "## Test Results
Details" section (regex fallback to full content if not found),
lowercase the status labels to "passed/failed", and move the
verification block into a collapsible <details> with the status shown in
the <summary>. In verify-tests-fail.ps1: downgrade top-level headings to
smaller heading levels (e.g. H1->H3, H2->H4) for Summary, Final Verdict,
Configuration, Test Results Details, and Logs to produce a
less-prominent, embeddable report. Changes are primarily formatting and
presentation to keep PR comments concise and readable.

---------

Co-authored-by: Shane Neuville <shane94@hotmail.com>
@kubaflo kubaflo added the copilot label Feb 6, 2026
This was referenced Feb 12, 2026
This was referenced Feb 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development copilot

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments