Skip to content

Add solution filtered parallel test run #4226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
westey-m merged 1 commit into from
Feb 24, 2026
Merged

Add solution filtered parallel test run #4226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .github/workflows/New-FrameworkFilteredSolution.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/usr/bin/env pwsh
# Copyright (c) Microsoft. All rights reserved.

<#
.SYNOPSIS
Generates a filtered .slnx solution file that only includes test projects supporting a given target framework.

.DESCRIPTION
Parses a .slnx solution file and queries each test project's TargetFrameworks using MSBuild.
Removes test projects that don't support the specified target framework, writes the result
to a temporary or specified output path, and prints the output path.

This is useful for running `dotnet test --solution` with MTP (Microsoft Testing Platform),
which requires all test projects in the solution to support the requested target framework.

.PARAMETER Solution
Path to the source .slnx solution file.

.PARAMETER TargetFramework
The target framework to filter by (e.g., net10.0, net472).

.PARAMETER Configuration
Optional MSBuild configuration used when querying TargetFrameworks. Defaults to Debug.

.PARAMETER OutputPath
Optional output path for the filtered .slnx file. If not specified, a temp file is created.

.EXAMPLE
# Generate a filtered solution and run tests
$filtered = ./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472
dotnet test --solution $filtered --no-build -f net472

.EXAMPLE
# Inline usage with dotnet test (PowerShell)
dotnet test --solution (./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472
Comment on lines +30 to +35
Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example paths in the script documentation reference "./eng/New-FilteredSolution.ps1", but the script is actually located at ".github/workflows/New-FrameworkFilteredSolution.ps1". This inconsistency between the documentation and the actual file location could confuse users trying to use the script.

Update the example paths in lines 30 and 35 to reflect the actual location: ".github/workflows/New-FrameworkFilteredSolution.ps1".

Suggested change
$filtered = ./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472
dotnet test --solution $filtered --no-build -f net472
.EXAMPLE
# Inline usage with dotnet test (PowerShell)
dotnet test --solution (./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472
$filtered = .github/workflows/New-FrameworkFilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472
dotnet test --solution $filtered --no-build -f net472
.EXAMPLE
# Inline usage with dotnet test (PowerShell)
dotnet test --solution (.github/workflows/New-FrameworkFilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472

Copilot uses AI. Check for mistakes.
#>

[CmdletBinding()]
param(
[Parameter(Mandatory)]
[string]$Solution,

[Parameter(Mandatory)]
[string]$TargetFramework,

[string]$Configuration = "Debug",

[string]$OutputPath
)

$ErrorActionPreference = "Stop"

# Resolve the solution path
$solutionPath = Resolve-Path $Solution
$solutionDir = Split-Path $solutionPath -Parent

if (-not $OutputPath) {
$OutputPath = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), "filtered-$(Split-Path $solutionPath -Leaf)")
}

# Parse the .slnx XML
[xml]$slnx = Get-Content $solutionPath -Raw

# Find all Project elements with paths containing "tests/"
$testProjects = $slnx.SelectNodes("//Project[contains(@Path, 'tests/')]")

$removed = @()
$kept = @()

foreach ($proj in $testProjects) {
$projRelPath = $proj.GetAttribute("Path")
$projFullPath = Join-Path $solutionDir $projRelPath

if (-not (Test-Path $projFullPath)) {
Write-Verbose "Project not found, keeping in solution: $projRelPath"
$kept += $projRelPath
continue
}

# Query the project's target frameworks using MSBuild
$targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null
Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dotnet msbuild command's stderr output is being redirected to $null (2>$null), which could hide important error messages about project parsing issues. If the project file is malformed or has build errors, these errors would be silently suppressed.

Consider capturing and logging stderr output, or at least preserving warnings while suppressing only expected informational messages. This would help diagnose issues when the script doesn't work as expected.

Suggested change
$targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null
$targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo

Copilot uses AI. Check for mistakes.
$targetFrameworks = $targetFrameworks.Trim()

if ($targetFrameworks -like "*$TargetFramework*") {
Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script uses substring matching with the -like operator on line 84 to check if a target framework is supported. While this works correctly for the framework monikers used in this repository (net10.0, net9.0, net8.0, net472), it could cause issues with platform-specific target frameworks in the future.

For example, if a project targets "net8.0-windows" and you filter for "net8.0", the substring match would succeed even though "net8.0" doesn't match "net8.0-windows" exactly. Consider splitting TargetFrameworks by semicolons and checking for exact matches to make the script more robust for future use cases.

Suggested change
if ($targetFrameworks -like "*$TargetFramework*") {
# TargetFrameworks is a semicolon-separated list; match the requested framework exactly
$frameworkList = $targetFrameworks -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne '' }
if ($frameworkList -contains $TargetFramework) {

Copilot uses AI. Check for mistakes.
Write-Verbose "Keeping: $projRelPath (targets: $targetFrameworks)"
$kept += $projRelPath
}
else {
Write-Verbose "Removing: $projRelPath (targets: $targetFrameworks, missing: $TargetFramework)"
$removed += $projRelPath
$proj.ParentNode.RemoveChild($proj) | Out-Null
}
}

# Write the filtered solution
$slnx.Save($OutputPath)
Comment on lines +91 to +96
Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script sets $ErrorActionPreference = "Stop" on line 51, which will cause the script to terminate on any error. However, when the script removes projects from the XML (line 91), it pipes to Out-Null. If RemoveChild fails for any reason, the script will stop without saving the filtered solution or providing a clear error message about which project caused the issue.

Consider adding try-catch blocks around critical operations (especially the XML manipulation and Save operations) to provide more meaningful error messages and ensure cleanup happens even if errors occur.

Suggested change
$proj.ParentNode.RemoveChild($proj) | Out-Null
}
}
# Write the filtered solution
$slnx.Save($OutputPath)
try {
$proj.ParentNode.RemoveChild($proj) | Out-Null
}
catch {
Write-Error "Failed to remove project '$projRelPath' from solution XML. Error: $($_.Exception.Message)"
}
}
}
# Write the filtered solution
try {
$slnx.Save($OutputPath)
}
catch {
Write-Error "Failed to save filtered solution to '$OutputPath'. Error: $($_.Exception.Message)"
throw
}

Copilot uses AI. Check for mistakes.

# Report results to stderr so stdout is clean for piping
Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment on line 98 states "Report results to stderr so stdout is clean for piping", but Write-Host doesn't write to stderr. In PowerShell 5.0 and later, Write-Host writes to the information stream (stream 6), not stderr (stream 2). In earlier versions, it writes directly to the console host.

If the intent is to write to stderr to keep stdout clean, use Write-Error (for errors) or redirect to stderr explicitly using Write-Output "..." | Out-File -FilePath ([Console]::Error) -Append. However, since Write-Host output doesn't interfere with variable assignment or piping (as shown in the examples on lines 30-35), the current implementation works correctly for its intended purpose. The comment should be updated to clarify that Write-Host writes to the information stream, not stderr.

Suggested change
# Report results to stderr so stdout is clean for piping
# Report results via Write-Host (information stream) so stdout is clean for piping

Copilot uses AI. Check for mistakes.
Write-Host "Filtered solution written to: $OutputPath" -ForegroundColor Green
if ($removed.Count -gt 0) {
Write-Host "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" -ForegroundColor Yellow
foreach ($r in $removed) {
Write-Host " - $r" -ForegroundColor Yellow
}
}
Write-Host "Kept $($kept.Count) test project(s)." -ForegroundColor Green
Comment on lines +99 to +106
Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script is called with the -Verbose flag on line 151, which will output verbose messages from the script. However, the script uses Write-Host (lines 99-106) for its main output instead of Write-Verbose. This means the verbose flag won't control whether these messages are displayed - they'll always be shown regardless of the -Verbose setting.

For proper verbose handling, informational messages about which projects are being kept or removed should use Write-Verbose instead of Write-Host, and the current Write-Verbose calls (lines 75, 85, 89) should remain as-is. This would allow the -Verbose flag to properly control the verbosity of the script's output.

Suggested change
Write-Host "Filtered solution written to: $OutputPath" -ForegroundColor Green
if ($removed.Count -gt 0) {
Write-Host "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" -ForegroundColor Yellow
foreach ($r in $removed) {
Write-Host " - $r" -ForegroundColor Yellow
}
}
Write-Host "Kept $($kept.Count) test project(s)." -ForegroundColor Green
Write-Verbose "Filtered solution written to: $OutputPath"
if ($removed.Count -gt 0) {
Write-Verbose "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:"
foreach ($r in $removed) {
Write-Verbose " - $r"
}
}
Write-Verbose "Kept $($kept.Count) test project(s)."

Copilot uses AI. Check for mistakes.

# Output the path for piping
Write-Output $OutputPath
71 changes: 38 additions & 33 deletions .github/workflows/dotnet-build-and-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,27 +140,37 @@ jobs:
popd
rm -rf "$TEMP_DIR"

- name: Generate filtered solution
shell: pwsh
run: |
.github/workflows/New-FrameworkFilteredSolution.ps1 `
-Solution dotnet/agent-framework-dotnet.slnx `
-TargetFramework ${{ matrix.targetFramework }} `
-Configuration ${{ matrix.configuration }} `
-OutputPath dotnet/filtered.slnx `
-Verbose

- name: Run Unit Tests
shell: bash
shell: pwsh
run: |
cd dotnet
export UT_PROJECTS=$(find ./ -type f -name "*.UnitTests.csproj" | tr '\n' ' ')
for project in $UT_PROJECTS; do
# Query the project's target frameworks using MSBuild with the current configuration
target_frameworks=$(dotnet msbuild $project -getProperty:TargetFrameworks -p:Configuration=${{ matrix.configuration }} -nologo 2>/dev/null | tr -d '\r')

# Check if the project supports the target framework
if [[ "$target_frameworks" == *"${{ matrix.targetFramework }}"* ]]; then
if [[ "${{ matrix.targetFramework }}" == "${{ env.COVERAGE_FRAMEWORK }}" ]]; then
# --ignore-exit-code 8: ignore failures due to finding no matching tests to run in a single project.
dotnet test --project $project -f ${{ matrix.targetFramework }} -c ${{ matrix.configuration }} --no-build -v Normal --report-xunit-trx --ignore-exit-code 8 --coverage --coverage-output-format cobertura --coverage-settings "$(pwd)/tests/coverage.runsettings" --results-directory "$(pwd)/../TestResults/Coverage/"
else
dotnet test --project $project -f ${{ matrix.targetFramework }} -c ${{ matrix.configuration }} --no-build -v Normal --report-xunit-trx --ignore-exit-code 8
fi
else
echo "Skipping $project - does not support target framework ${{ matrix.targetFramework }} (supports: $target_frameworks)"
fi
done
$coverageArgs = @()
if ("${{ matrix.targetFramework }}" -eq "${{ env.COVERAGE_FRAMEWORK }}") {
$coverageArgs = @(
"--coverage",
"--coverage-output-format", "cobertura",
"--coverage-settings", "dotnet/tests/coverage.runsettings",
"--results-directory", "TestResults/Coverage/"
)
}

dotnet test --solution dotnet/filtered.slnx `
-f ${{ matrix.targetFramework }} `
-c ${{ matrix.configuration }} `
--no-build -v Normal `
--report-xunit-trx `
--ignore-exit-code 8 `
--filter-query "/*UnitTests*/*/*/*" `
@coverageArgs
env:
# Cosmos DB Emulator connection settings
COSMOSDB_ENDPOINT: https://localhost:8081
Expand All @@ -187,22 +197,17 @@ jobs:
id: azure-functions-setup

- name: Run Integration Tests
shell: bash
shell: pwsh
if: github.event_name != 'pull_request' && matrix.integration-tests
run: |
cd dotnet
export INTEGRATION_TEST_PROJECTS=$(find ./ -type f -name "*IntegrationTests.csproj" | tr '\n' ' ')
for project in $INTEGRATION_TEST_PROJECTS; do
# Query the project's target frameworks using MSBuild with the current configuration
target_frameworks=$(dotnet msbuild $project -getProperty:TargetFrameworks -p:Configuration=${{ matrix.configuration }} -nologo 2>/dev/null | tr -d '\r')

# Check if the project supports the target framework
if [[ "$target_frameworks" == *"${{ matrix.targetFramework }}"* ]]; then
dotnet test --project $project -f ${{ matrix.targetFramework }} -c ${{ matrix.configuration }} --no-build -v Normal --report-xunit-trx --ignore-exit-code 8 --filter-not-trait "Category=IntegrationDisabled"
else
echo "Skipping $project - does not support target framework ${{ matrix.targetFramework }} (supports: $target_frameworks)"
fi
done
dotnet test --solution dotnet/filtered.slnx `
-f ${{ matrix.targetFramework }} `
-c ${{ matrix.configuration }} `
--no-build -v Normal `
--report-xunit-trx `
--ignore-exit-code 8 `
--filter-query "/*IntegrationTests*/*/*/*" `
--filter-not-trait "Category=IntegrationDisabled"
env:
# Cosmos DB Emulator connection settings
COSMOSDB_ENDPOINT: https://localhost:8081
Expand Down