Add solution filtered parallel test run#4226
Add solution filtered parallel test run#4226westey-m merged 1 commit intomicrosoft:feature-xunit3-mtp-upgradefrom
Conversation
a39e324
into
microsoft:feature-xunit3-mtp-upgrade
There was a problem hiding this comment.
Pull request overview
This PR migrates the .NET test execution from sequential bash-based iteration to parallel execution using Microsoft Testing Platform (MTP). A new PowerShell script filters solution files by target framework, enabling dotnet test --solution to run all tests in parallel across supported projects.
Changes:
- Added a PowerShell script to dynamically filter .slnx solution files by target framework, removing test projects that don't support the specified framework
- Replaced bash-based sequential test execution (iterating through projects) with PowerShell-based parallel test execution using filtered solutions and
--filter-query - Maintained all existing test filtering, coverage collection, and environment configuration while improving performance through parallelization
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| .github/workflows/New-FrameworkFilteredSolution.ps1 | New PowerShell script that filters .slnx solution files to include only test projects supporting a specific target framework |
| .github/workflows/dotnet-build-and-test.yml | Migrated unit and integration test execution from bash to PowerShell, using filtered solutions for parallel test execution with --filter-query patterns |
| # Write the filtered solution | ||
| $slnx.Save($OutputPath) | ||
|
|
||
| # Report results to stderr so stdout is clean for piping |
There was a problem hiding this comment.
The comment on line 98 states "Report results to stderr so stdout is clean for piping", but Write-Host doesn't write to stderr. In PowerShell 5.0 and later, Write-Host writes to the information stream (stream 6), not stderr (stream 2). In earlier versions, it writes directly to the console host.
If the intent is to write to stderr to keep stdout clean, use Write-Error (for errors) or redirect to stderr explicitly using Write-Output "..." | Out-File -FilePath ([Console]::Error) -Append. However, since Write-Host output doesn't interfere with variable assignment or piping (as shown in the examples on lines 30-35), the current implementation works correctly for its intended purpose. The comment should be updated to clarify that Write-Host writes to the information stream, not stderr.
| # Report results to stderr so stdout is clean for piping | |
| # Report results via Write-Host (information stream) so stdout is clean for piping |
| } | ||
|
|
||
| # Query the project's target frameworks using MSBuild | ||
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null |
There was a problem hiding this comment.
The dotnet msbuild command's stderr output is being redirected to $null (2>$null), which could hide important error messages about project parsing issues. If the project file is malformed or has build errors, these errors would be silently suppressed.
Consider capturing and logging stderr output, or at least preserving warnings while suppressing only expected informational messages. This would help diagnose issues when the script doesn't work as expected.
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null | |
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo |
| $filtered = ./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472 | ||
| dotnet test --solution $filtered --no-build -f net472 | ||
|
|
||
| .EXAMPLE | ||
| # Inline usage with dotnet test (PowerShell) | ||
| dotnet test --solution (./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472 |
There was a problem hiding this comment.
The example paths in the script documentation reference "./eng/New-FilteredSolution.ps1", but the script is actually located at ".github/workflows/New-FrameworkFilteredSolution.ps1". This inconsistency between the documentation and the actual file location could confuse users trying to use the script.
Update the example paths in lines 30 and 35 to reflect the actual location: ".github/workflows/New-FrameworkFilteredSolution.ps1".
| $filtered = ./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472 | |
| dotnet test --solution $filtered --no-build -f net472 | |
| .EXAMPLE | |
| # Inline usage with dotnet test (PowerShell) | |
| dotnet test --solution (./eng/New-FilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472 | |
| $filtered = .github/workflows/New-FrameworkFilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472 | |
| dotnet test --solution $filtered --no-build -f net472 | |
| .EXAMPLE | |
| # Inline usage with dotnet test (PowerShell) | |
| dotnet test --solution (.github/workflows/New-FrameworkFilteredSolution.ps1 -Solution ./agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472 |
| Write-Host "Filtered solution written to: $OutputPath" -ForegroundColor Green | ||
| if ($removed.Count -gt 0) { | ||
| Write-Host "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" -ForegroundColor Yellow | ||
| foreach ($r in $removed) { | ||
| Write-Host " - $r" -ForegroundColor Yellow | ||
| } | ||
| } | ||
| Write-Host "Kept $($kept.Count) test project(s)." -ForegroundColor Green |
There was a problem hiding this comment.
The script is called with the -Verbose flag on line 151, which will output verbose messages from the script. However, the script uses Write-Host (lines 99-106) for its main output instead of Write-Verbose. This means the verbose flag won't control whether these messages are displayed - they'll always be shown regardless of the -Verbose setting.
For proper verbose handling, informational messages about which projects are being kept or removed should use Write-Verbose instead of Write-Host, and the current Write-Verbose calls (lines 75, 85, 89) should remain as-is. This would allow the -Verbose flag to properly control the verbosity of the script's output.
| Write-Host "Filtered solution written to: $OutputPath" -ForegroundColor Green | |
| if ($removed.Count -gt 0) { | |
| Write-Host "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" -ForegroundColor Yellow | |
| foreach ($r in $removed) { | |
| Write-Host " - $r" -ForegroundColor Yellow | |
| } | |
| } | |
| Write-Host "Kept $($kept.Count) test project(s)." -ForegroundColor Green | |
| Write-Verbose "Filtered solution written to: $OutputPath" | |
| if ($removed.Count -gt 0) { | |
| Write-Verbose "Removed $($removed.Count) test project(s) not targeting ${TargetFramework}:" | |
| foreach ($r in $removed) { | |
| Write-Verbose " - $r" | |
| } | |
| } | |
| Write-Verbose "Kept $($kept.Count) test project(s)." |
| $proj.ParentNode.RemoveChild($proj) | Out-Null | ||
| } | ||
| } | ||
|
|
||
| # Write the filtered solution | ||
| $slnx.Save($OutputPath) |
There was a problem hiding this comment.
The script sets $ErrorActionPreference = "Stop" on line 51, which will cause the script to terminate on any error. However, when the script removes projects from the XML (line 91), it pipes to Out-Null. If RemoveChild fails for any reason, the script will stop without saving the filtered solution or providing a clear error message about which project caused the issue.
Consider adding try-catch blocks around critical operations (especially the XML manipulation and Save operations) to provide more meaningful error messages and ensure cleanup happens even if errors occur.
| $proj.ParentNode.RemoveChild($proj) | Out-Null | |
| } | |
| } | |
| # Write the filtered solution | |
| $slnx.Save($OutputPath) | |
| try { | |
| $proj.ParentNode.RemoveChild($proj) | Out-Null | |
| } | |
| catch { | |
| Write-Error "Failed to remove project '$projRelPath' from solution XML. Error: $($_.Exception.Message)" | |
| } | |
| } | |
| } | |
| # Write the filtered solution | |
| try { | |
| $slnx.Save($OutputPath) | |
| } | |
| catch { | |
| Write-Error "Failed to save filtered solution to '$OutputPath'. Error: $($_.Exception.Message)" | |
| throw | |
| } |
| $targetFrameworks = & dotnet msbuild $projFullPath -getProperty:TargetFrameworks -p:Configuration=$Configuration -nologo 2>$null | ||
| $targetFrameworks = $targetFrameworks.Trim() | ||
|
|
||
| if ($targetFrameworks -like "*$TargetFramework*") { |
There was a problem hiding this comment.
The script uses substring matching with the -like operator on line 84 to check if a target framework is supported. While this works correctly for the framework monikers used in this repository (net10.0, net9.0, net8.0, net472), it could cause issues with platform-specific target frameworks in the future.
For example, if a project targets "net8.0-windows" and you filter for "net8.0", the substring match would succeed even though "net8.0" doesn't match "net8.0-windows" exactly. Consider splitting TargetFrameworks by semicolons and checking for exact matches to make the script more robust for future use cases.
| if ($targetFrameworks -like "*$TargetFramework*") { | |
| # TargetFrameworks is a semicolon-separated list; match the requested framework exactly | |
| $frameworkList = $targetFrameworks -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne '' } | |
| if ($frameworkList -contains $TargetFramework) { |
* .NET: Upgrade to XUnit 3 and Microsoft Testing Platform (#4176) * Fix copilot studio integration tests failure (#4209) * Fix anthropic integration tests and skip reason (#4211) * Remove accidental add of code coverage for integration tests (#4219) * Add solution filtered parallel test run (#4226) * Fix build paths (#4228) * Fix coverage settings path and trait filter (#4229) * Add project name filter to solution (#4231) * Increase Integration Test Parallelism (#4241) * Increase integration tests threads to 4x (#4242) * Separate build and test into parallel jobs (#4243) * Filter src by framework for tests build (#4244) * Separate build and test into parallel jobs * Filter source projects by framework for tests build * Pre-build samples via tests to avoid timeouts (#4245) * Separate build from run for console sample validation (#4251) * Address PR comments (#4255) * Merge and move scripts (#4308) * .NET: Add Microsoft Fabric sample #3674 (#4230) Co-authored-by: Chris <66376200+crickman@users.noreply.github.com> * Python: Phase 2: Embedding clients for Ollama, Bedrock, and Azure AI Inference (#4207) * Phase 2: Embedding clients for Ollama, Bedrock, and Azure AI Inference Add embedding client implementations to existing provider packages: - OllamaEmbeddingClient: Text embeddings via Ollama's embed API - BedrockEmbeddingClient: Text embeddings via Amazon Titan on Bedrock - AzureAIInferenceEmbeddingClient: Text and image embeddings via Azure AI Inference, supporting Content | str input with separate model IDs for text (AZURE_AI_INFERENCE_EMBEDDING_MODEL_ID) and image (AZURE_AI_INFERENCE_IMAGE_EMBEDDING_MODEL_ID) endpoints Additional changes: - Rename EmbeddingCoT -> EmbeddingT, EmbeddingOptionsCoT -> EmbeddingOptionsT - Add otel_provider_name passthrough to all embedding clients - Register integration pytest marker in all packages - Add lazy-loading namespace exports for Ollama and Bedrock embeddings - Add image embedding sample using Cohere-embed-v3-english - Add azure-ai-inference dependency to azure-ai package Part of #1188 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix mypy duplicate name and ruff lint issues - Rename second 'vector' variable to 'img_vector' in image embedding loop - Combine nested with statements in tests - Remove unused result assignments in tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updates from feedback * Fix CI failures in embedding usage handling - Fix Azure AI embedding mypy issues by normalizing vectors to list[float], safely accumulating optional usage token fields, and filtering None entries before constructing GeneratedEmbeddings - Avoid Bandit false positive by initializing usage details as an empty dict - Update OpenAI embedding tests to assert canonical usage keys (input_token_count/total_token_count) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Purview] Mark responses as responses and fix epoch bug for python long overflow (#4225) * .NET: Support InvokeMcpTool for declarative workflows (#4204) * Initial implementation of InvokeMcpTool in declarative workflow * Cleaned up sample implementation * Updated sample comments. * Added missing executor routing attribute * Fix PR comments. * Updated based on PR comments. * Updated based on PR comments. * Removed unnecessary using statement. * Update Python package versions to rc2 (#4258) - Bump core and azure-ai to 1.0.0rc2 - Bump preview packages to 1.0.0b260225 - Update dependencies to >=1.0.0rc2 - Add CHANGELOG entries for changes since rc1 - Update uv.lock Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: Fixing issue where OpenTelemetry span is never exported in .NET in-process workflow execution (#4196) * 1. Add reproduction test for issue #4155: workflow.run Activity never stopped in streaming OffThread path The WorkflowRunActivity_IsStopped_Streaming_OffThread test demonstrates that the workflow.run OpenTelemetry Activity created in StreamingRunEventStream.RunLoopAsync is started but never stopped when using the OffThread/Default streaming execution. The background run loop keeps running after event consumption completes, so the using Activity? declaration never disposes until explicit StopAsync() is called. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> 2. Fix workflow.run Activity never stopped in streaming OffThread execution (#4155) The workflow.run OpenTelemetry Activity in StreamingRunEventStream.RunLoopAsync was scoped to the method lifetime via 'using'. Since the run loop only exits on cancellation, the Activity was never stopped/exported until explicit disposal. Fix: Remove 'using' and explicitly dispose the Activity when the workflow reaches Idle status (all supersteps complete). A safety-net disposal in the finally block handles cancellation and error paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add root-level workflow.session activity spanning run loop lifetime\n\nImplements two-level telemetry hierarchy per PR feedback from lokitoth:\n- workflow.session: spans the entire run loop / stream lifetime\n- workflow_invoke: per input-to-halt cycle, nested within the session\n\nThis ensures the session activity stays open across multiple turns,\nwhile individual run activities are created and disposed per cycle.\n\nAlso fixes linkedSource CancellationTokenSource disposal leak in\nStreamingRunEventStream (added using declaration)." * Address Copilot review: fix Activity/CTS disposal, rename activity, add error tag\n\n1. LockstepRunEventStream: Remove 'using' from Activity in async iterator\n and manually dispose in finally block (fixes #4155 pattern). Also dispose\n linkedSource CTS in finally to prevent leak.\n2. Tags.cs: Add ErrorMessage (\"error.message\") tag for runtime errors,\n distinct from BuildErrorMessage (\"build.error.message\").\n3. ActivityNames: Rename WorkflowRun from \"workflow_invoke\" to \"workflow.run\"\n for cross-language consistency.\n4. WorkflowTelemetryContext: Fix XML doc to say \"outer/parent span\" instead\n of \"root-level span\".\n5. ObservabilityTests: Assert WorkflowSession absence when DisableWorkflowRun\n is true.\n6. WorkflowRunActivityStopTests: Fix streaming test race by disposing\n StreamingRun before asserting activities are stopped.\n7. StreamingRunEventStream/LockstepRunEventStream: Use Tags.ErrorMessage\n instead of Tags.BuildErrorMessage for runtime error events." * Review fixes: revert workflow_invoke rename, use 'using' for linkedSource, move SessionStarted earlier\n\n- Revert ActivityNames.WorkflowRun back to \"workflow_invoke\" (OTEL semantic convention contract)\n- Use 'using' declaration for linkedSource CTS in LockstepRunEventStream (no timing sensitivity)\n- Move SessionStarted event before WaitForInputAsync in StreamingRunEventStream to match Lockstep behavior" * Improve naming and comments in WorkflowRunActivityStopTests" * Prevent session Activity.Current leak in lockstep mode, add nesting test Save and restore Activity.Current in LockstepRunEventStream.Start() so the session activity doesn't leak into caller code via AsyncLocal. Re-establish Activity.Current = sessionActivity before creating the run activity in TakeEventStreamAsync to preserve parent-child nesting. Add test verifying app activities after RunAsync are not parented under the session, and that the workflow_invoke activity nests under the session." * Fix stale XML doc: WorkflowRun -> WorkflowInvoke in ObservabilityTests --------- Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Python / .NET Samples - Restructure and Improve Samples (Feature Branc… (#4092) * Python: .NET Samples - Restructure and Improve Samples (Feature Branch) (#4091) * Moved by agent (#4094) * Fix readme links * .NET Samples - Create `04-hosting` learning path step (#4098) * Agent move * Agent reorderd * Remove A2A section from README Removed A2A section from the Getting Started README. * Agent fixed links * Fix broken sample links in durable-agents README (#4101) * Initial plan * Fix broken internal links in documentation Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> * Revert template link changes; keep only durable-agents README fix Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> * .NET Samples - Create `03-workflows` learning path step (#4102) * Fix solution project path * Python: Fix broken markdown links to repo resources (outside /docs) (#4105) * Initial plan * Fix broken markdown links to repo resources Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> * Update README to rename .NET Workflows Samples section --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> * .NET Samples - Create `02-agents` learning path step (#4107) * .NET: Fix broken relative link in GroupChatToolApproval README (#4108) * Initial plan * Fix broken link in GroupChatToolApproval README Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> * Update labeler configuration for workflow samples * .NET - Reorder Agents samples to start from Step01 instead of Step04 (#4110) * Fix solution * Resolve new sample paths * Move new AgentSkills and AgentWithMemory_Step04 samples * Fix link * Fix readme path * fix: update stale dotnet/samples/Durable path reference in AGENTS.md Co-authored-by: crickman <66376200+crickman@users.noreply.github.com> * Moved new sample * Update solution * Resolve merge (new sample) * Sync to new sample - FoundryAgents_Step21_BingCustomSearch * Updated README * .NET Samples - Configuration Naming Update (#4149) * .NET: Restore AzureFunctions index parity with ConsoleApps under DurableAgents samples (#4221) * Clean-up `05_host_your_agent` * Config setting consistency * Refine samples * AGENTS.md * Move new samples * Re-order samples * Move new project and fixup solution * Fixup model config * Fix up new UT project --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> * Python: Fix Bedrock embedding test stub missing meta attribute (#4287) * Fix Bedrock embedding test stub missing meta attribute * Increase test coverage so gate passes * Python: (ag-ui): fix approval payloads being re-processed on subsequent conversation turns (#4232) * Fix ag-ui tool call issue * Safe json fix * Python: Update workflow orchestration samples to use AzureOpenAIResponsesClient (#4285) * Update workflow orchestration samples to use AzureOpenAIResponsesClient * Fix broken link * Move scripts to scripts folder --------- Co-authored-by: Roger Barreto <19890735+rogerbarreto@users.noreply.github.com> Co-authored-by: Chris <66376200+crickman@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Rishabh Chawla <rishabhchawla1995@gmail.com> Co-authored-by: Peter Ibekwe <109177538+peibekwe@users.noreply.github.com> Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com> Co-authored-by: Ben Thomas <ben.thomas@microsoft.com> Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com> * Fix encoding (#4309) * Disable Parallelization for WorkflowRunActivityStopTests (#4313) * Revert parallel disable (#4324) * .NET: Disable flakey Workflow Observability tests (#4416) * Disable flakey OffThread test * Disable additional OffThread test * Disable a further test * Disable all observability tests --------- Co-authored-by: Roger Barreto <19890735+rogerbarreto@users.noreply.github.com> Co-authored-by: Chris <66376200+crickman@users.noreply.github.com> Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Rishabh Chawla <rishabhchawla1995@gmail.com> Co-authored-by: Peter Ibekwe <109177538+peibekwe@users.noreply.github.com> Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com> Co-authored-by: Ben Thomas <ben.thomas@microsoft.com> Co-authored-by: alliscode <bentho@microsoft.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
Motivation and Context
Description
Contribution Checklist