[CI] Generate simple test summary by radical · Pull Request #9223 · microsoft/aspire

radical · 2025-05-10T03:36:27Z

Usage: dotnet tools run GenerateTestSummary --dirPathOrTrxFilePath <path> [--output <output>] [--combined]

Generates a summary report from trx files.
- And writes to $GITHUB_STEP_SUMMARY if running in GitHub Actions
This can generate individual summary for a test assembly showing details of the test failures.
And a combined report for multiple test assemblies with the total counters shown.

davidfowl · 2025-05-10T04:18:00Z

@radical can you translate this to a C# project under a tools folder? We don't need this to be in powershell 😄 (and it'll let us all vibe code a bit better on it)

radical · 2025-05-10T04:25:34Z

@radical can you translate this to a C# project under a tools folder? We don't need this to be in powershell 😄 (and it'll let us all vibe code a bit better on it)

This is essentially a stopgap till @RussKie and @joperezr have the other test reporter back again. But we could do this as our own thing in C# too. Copilot seems to be good with powershell too btw. That's what I am using right now lol.

danmoseley · 2025-05-10T17:19:20Z

Copilot seems to be good with powershell too btw.

pick your AI, you'll find one that can easily translate that to C#

davidfowl · 2025-05-10T17:20:33Z

You can stop gap in c#

danmoseley · 2025-05-10T22:13:49Z

Actually thinking about it -- if psh is on all the test machines, isn't it the best tool for the job? No compilation and easy to read/modify (even debug).. but idc

danmoseley · 2025-05-10T22:19:07Z

/home/runner/work/aspire/aspire/dotnet.sh ... looks like a workaround may be to use $GITHUB_WORKSPACE instead?
actions/checkout#785 (comment)

davidfowl · 2025-05-11T18:04:02Z

Can you inline the logs for the failed test under a nested collapsible section? Like

Details

in markdown

RussKie · 2025-05-12T01:56:38Z

Have you explored the MTP extensibility model or chatted with the Test team? The MTP supports custom plugins that can be invoked at specific points in the test execution workflow.

On a heel of #8811 (comment), I had a chat with @Youssef1313, here's some details from that convo:

Registration of extensions happens in the auto-generated entry point. We use an MSBuild item for that (named TestingPlatformBuilderHook)
  <ItemGroup>
    <TestingPlatformBuilderHook Include="UNIQUE_GUID_FOR_YOUR_EXTENSION" >
      <DisplayName>Display Name for your extension</DisplayName>
      <TypeFullName>FQN of TestingPlatformBuilderHook</TypeFullName>
    </TestingPlatformBuilderHook>
  </ItemGroup>
The TestingPlatformBuilderHook class should have AddExtensions method with two parameters (ITestApplicationBuilder, and string[]). The latter in most cases ends up unused (it represents the command-line arguments, i.e, same as string[] args of Main)
On the ITestApplicationBuilder, you do stuff like builder.TestHost.AddDataConsumer as explained in https://learn.microsoft.com/en-us/dotnet/core/testing/microsoft-testing-platform-architecture-extensions#the-idataconsumer-extensions.

See https://github.com/microsoft/testfx/blob/5bcd432d2e9289609395ab2ae1f8489ec9ed9610/samples/public/mstest-runner/CustomReportExtension/CustomReportExtension/Program.cs#L12-L19 for example. The code in this sample is what you can have in AddExtensions.

@nohwnd also had a presentation for an AzDO reporting plugin during the latest .NET Showcase. Jakub also told me that a GHA reported was something on the team's backlog.

Last, but not least @jeffhandley has built an awesome reporting for https://github.com/dotnet/issue-labeler (dotnet/issue-labeler#104) using GitHub.Actions.Core (which is built by our own @IEvangelist).
You can see the examples in our actions. E.g.:

With the above, I'm not entirely convinced this change may be going the right direction; we'll definitely need the data analysis and the reporting capabilities, but the overall strategy is worth reconsidering.

radical · 2025-05-12T02:05:17Z

With the above, I'm not entirely convinced this change may be going the right direction; we'll definitely need the data analysis and the reporting capabilities, but the overall strategy is worth reconsidering.

Thank you, that's useful information! I'm aware that you have great ideas around this and are going to possibly work on it. But that might take some time, so my effort here is a just a quick solution to act as a stop-gap till your proper solution is merged.

davidfowl · 2025-05-12T04:33:35Z

Yea, lets do that in a follow up PR. We just need something better than what we have now at the moment.

radical · 2025-05-12T04:35:09Z

radical · 2025-05-12T04:36:02Z

test run - https://github.com/dotnet/aspire/actions/runs/14963668924?pr=9223

Just cleaning up the PR now.

davidfowl · 2025-05-12T04:37:34Z

Super nice. Small nit, delete the Program class and static void Main. Top level statements are made for these micro level programs 😄

davidfowl · 2025-05-12T04:45:41Z

@radical can we see inline console logs for the test that failed? Where is that?

radical · 2025-05-12T04:47:26Z

@radical can we see inline console logs for the test that failed? Where is that?

That should be captured in the .trx file. I am trying to surface that here too though, but the github actions summary has a limit of 64k. You can download the trx and log files from the artifacts named like logs-Hosting-ubuntu-latest.

davidfowl · 2025-05-12T04:54:14Z

OK some suggestions, add a link to the TRX file and logs per test failure. That will make it trivial to find and download the logs instead of showing them inline.

RussKie · 2025-05-12T05:10:59Z

These look awesome!

radical · 2025-05-12T05:18:58Z

OK some suggestions, add a link to the TRX file and logs per test failure. That will make it trivial to find and download the logs

Added link to the artifacts for the test assembly, which includes the trx, log file etc.

instead of showing them inline.

I retained this one though, good for a quick look at the error.

Copilot

Pull Request Overview

This PR adds a new tool for generating test summary reports from trx files, including both combined and single reports, and integrates it into the CI workflows.

Introduces a new project, GenerateTestSummary, with logic for reading trx files and creating markdown reports.
Updates GitHub workflows to use the new tool for generating test result summaries.
Integrates the new project into the solution.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/GenerateTestSummary/TrxReader.cs	Adds functionality to deserialize and process trx files into test results.
tools/GenerateTestSummary/TestSummaryGenerator.cs	Implements report generation logic for both combined and single test summaries.
tools/GenerateTestSummary/Program.cs	Provides command-line entry point to run the test summary generation.
tools/GenerateTestSummary/GenerateTestSummary.csproj	New project file setup for the test summary tool.
Aspire.sln	Integrates the new project into the solution.
.github/workflows/tests.yml	Updates CI steps to use the new test summary tool with absolute paths.
.github/workflows/run-tests.yml	Adds a step to generate test summaries and pass URLs for logs.

tools/GenerateTestSummary/TestSummaryGenerator.cs

Copilot · 2025-05-12T05:21:47Z

tools/GenerateTestSummary/TestSummaryGenerator.cs

+                errorMsgBuilder.AppendLine(test.Output?.ErrorInfo?.InnerText ?? string.Empty);
+                errorMsgBuilder.AppendLine(test.Output?.StdOut ?? string.Empty);
+
+                var errorMsgTruncated = TruncateTheStart(errorMsgBuilder.ToString(), 50_000); // Truncate long error messages for readability


[nitpick] Consider refactoring the magic number 50_000 into a named constant to clarify its purpose and improve maintainability.

davidfowl · 2025-05-12T05:22:34Z

@radical this PR is good, extra credit for a follow up PR that posts a GitHub comment with the direct link to the summary 😄 (and updates that same comment as re-runs happen)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

radical · 2025-05-12T05:26:50Z

@radical this PR is good, extra credit for a follow up PR that posts a GitHub comment with the direct link to the summary 😄 (and updates that same comment as re-runs happen)

I was trying to add the reports on the Checks page actually, which would surface that in the PR nicely too. But a quick attempt at it failed because of permissions to post from the workflow.

PS: I have a different tool that I am working on which would add a github app and we could post from that.

davidfowl · 2025-05-12T05:37:59Z

github app seems over engineered what what we need. Actions can use the token to post comments like our backport bot.

Anyhow, approved.

radical · 2025-05-12T05:48:41Z

Actions can use the token to post comments like our backport bot.

I'll check that out 👍

davidfowl · 2025-05-12T05:56:35Z

https://github.com/dotnet/arcade/blob/80c4e4d26cb85c86f7e1be77d2d9eceeef0f3493/.github/workflows/backport-base.yml#L62-L73

davidfowl · 2025-05-12T06:02:59Z

@radical include the OS in the test name. It looks like there are duplicate runs in the summary

RussKie · 2025-05-12T09:22:27Z

tools/GenerateTestSummary/Program.cs

+// Generate a summary report from trx files.
+// And write to $GITHUB_STEP_SUMMARY if running in GitHub Actions.
+
+var dirPathOrTrxFilePathArgument = new Argument<string>("dirPathOrTrxFilePath");


I'm struggling to understand all use-cases for this argument; more comments would be really helpful.

I understand when this points to a folder where *.trx files are, but what does it mean when it is a file? Why would I pass a file?

What does the following combination means?

-- D:\a\aspire\aspire/testresults -u https://github.com/dotnet/aspire/actions/runs/14966104757/artifacts/3103892024

github-actions bot added the area-engineering-systems infrastructure helix infra engineering repo stuff label May 10, 2025

[CI] Add a tools/GenerateTestSummary.csproj for use with github actions

edfca6e

radical force-pushed the test-summary branch from de38ea2 to edfca6e Compare May 12, 2025 04:46

radical marked this pull request as ready for review May 12, 2025 04:47

radical requested review from RussKie, danmoseley, davidfowl and joperezr May 12, 2025 04:49

Address feedback from @ davidfowler and add link to the logs

ef704aa

davidfowl requested a review from Copilot May 12, 2025 05:21

Copilot AI reviewed May 12, 2025

View reviewed changes

Update tools/GenerateTestSummary/TestSummaryGenerator.cs

827200d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

cleanup

524dd08

davidfowl approved these changes May 12, 2025

View reviewed changes

radical merged commit 34ada42 into microsoft:main May 12, 2025
170 checks passed

radical deleted the test-summary branch May 12, 2025 05:50

RussKie reviewed May 12, 2025

View reviewed changes

github-actions bot locked and limited conversation to collaborators Jun 12, 2025

Conversation

radical commented May 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidfowl commented May 10, 2025

Uh oh!

radical commented May 10, 2025

Uh oh!

danmoseley commented May 10, 2025

Uh oh!

davidfowl commented May 10, 2025

Uh oh!

danmoseley commented May 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danmoseley commented May 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidfowl commented May 11, 2025

Uh oh!

RussKie commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

RussKie commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI May 12, 2025

Choose a reason for hiding this comment

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

radical commented May 12, 2025

Uh oh!

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

davidfowl commented May 12, 2025

Uh oh!

RussKie May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

radical commented May 10, 2025 •

edited

Loading

danmoseley commented May 10, 2025 •

edited

Loading

danmoseley commented May 10, 2025 •

edited

Loading