Add E2E coverage for dotnet test (MTP) live test-host output - #55517
Merged
Conversation
Covers #51615: a test app's stdout/stderr must reach the user while the run is in progress, not only when a test fails. The new TestProjectWithLiveOutput asset writes to the console both before the pipe handshake completes and while the test session runs, and reports a single passing test. The test asserts all three markers appear in the dotnet test output of a fully successful run, and that the in-run output precedes the end-of-run summary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eedfe288-df4b-48f1-a612-67ce78b017a5
…path Addresses review feedback on the previous commit: - The ordering assertion on the captured stdout could not fail: the child's output streams are fully drained before the summary is emitted, so even a buffer-until-exit implementation satisfied it. Replaced with a real observation: the test app blocks until a sentinel file appears, and the test creates that file from CommandOutputHandler when it sees the marker on the live standard output of 'dotnet test'. Output that is not forwarded live therefore deadlocks the app until its own timeout expires and fails the run. - The pre-handshake marker did not guard the flush-on-negotiation path, because a later line flushed the whole buffer anyway. Moved it to a dedicated asset that writes only before the handshake and produces nothing afterwards. - Added coverage for output written through the platform's IOutputDevice, which reaches the SDK as protocol 1.3.0 display messages. Session, warning and error messages are covered; plain informational text is deliberately discarded by the host under the pipe protocol. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eedfe288-df4b-48f1-a612-67ce78b017a5
Two problems found reviewing the sentinel handshake: - Execute retries the command on transient failures without re-copying the test asset, so a sentinel created by an earlier attempt would still be there on the next one. The app would then return immediately and the test would go green having proven nothing. Clear the sentinel per attempt from ProcessStartedHandler, which runs right after the process starts. - The output handler runs on the only thread draining the command's standard output. An exception from the file write would stop that drain, so the child would block once the pipe buffer filled and the run would hang instead of failing. Capture the exception and assert on it after the run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eedfe288-df4b-48f1-a612-67ce78b017a5
The delete runs after the command's process has started but before it is registered with the process reaper, so an exception escaping it would leave that process orphaned. Record it like the write failure and assert on it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eedfe288-df4b-48f1-a612-67ce78b017a5
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end tests in dotnet.Tests to validate that dotnet test (Microsoft.Testing.Platform / MTP) forwards test-host output to the user during an in-progress run, including output written before protocol negotiation completes.
Changes:
- Added a new
dotnet.TestsE2E test (GivenDotnetTestForwardsTestHostOutput) covering (1) live forwarding while the run is still active and (2) buffering + flush of output produced before the handshake completes. - Added two new MTP test assets (
TestProjectWithLiveOutput,TestProjectWithOutputBeforeHandshake) to exercise console stdout/stderr and output-deviceDisplayMessagebehavior under the pipe protocol.
Show a summary per file
| File | Description |
|---|---|
| test/TestAssets/TestProjects/TestProjectWithOutputBeforeHandshake/TestProjectWithOutputBeforeHandshake.csproj | New MTP test-asset project to emit a single pre-handshake stdout line. |
| test/TestAssets/TestProjects/TestProjectWithOutputBeforeHandshake/Program.cs | Implements the minimal MTP app that prints before CreateBuilderAsync and then publishes a passing test node. |
| test/TestAssets/TestProjects/TestProjectWithOutputBeforeHandshake/global.json | Declares MTP as the test runner for the asset. |
| test/TestAssets/TestProjects/TestProjectWithLiveOutput/TestProjectWithLiveOutput.csproj | New MTP test-asset project to exercise live stdout/stderr and output-device forwarding. |
| test/TestAssets/TestProjects/TestProjectWithLiveOutput/Program.cs | Implements the sentinel-based “must be observed live” handshake and emits stdout/stderr + output-device markers. |
| test/TestAssets/TestProjects/TestProjectWithLiveOutput/global.json | Declares MTP as the test runner for the asset. |
| test/dotnet.Tests/CommandTests/Test/GivenDotnetTestForwardsTestHostOutput.cs | New E2E coverage validating live forwarding and pre-handshake buffering behavior through dotnet test. |
Copilot's findings
- Files reviewed: 7/7 changed files
- Comments generated: 1
Evangelink
enabled auto-merge
July 29, 2026 10:59
Evangelink
disabled auto-merge
July 29, 2026 10:59
Evangelink
enabled auto-merge
July 29, 2026 10:59
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8782400-17e4-4447-b71f-06288a76e89e
nohwnd
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #51615.
The behavior change itself shipped earlier (#54825 plus follow-ups: highest-common protocol negotiation, live stdout/stderr streaming gated on ≥ 1.1.0, and forwarding of
AzureDevOpsLogMessage(1.2.0) andDisplayMessage(1.3.0)). What was missing was end-to-end coverage: the existing tests were unit-level only (protocol negotiation + serializer round-trips), so nothing guarded the actual user-visible behavior throughdotnet test.What this adds
test/dotnet.Tests/CommandTests/Test/GivenDotnetTestForwardsTestHostOutput.cswith two scenarios, each run forDebugandRelease:1. Output is forwarded while the run is in progress —
TestProjectWithLiveOutputAsserting on captured text after the process exits cannot distinguish "streamed live" from "buffered until exit", so this uses a sentinel handshake instead:
CommandOutputHandlercreates that sentinel only when it sees the marker on the live output of the still-runningdotnet test;An implementation that buffers host output until process exit therefore deadlocks the app and fails, rather than passing by replaying everything just before the summary.
The same asset also exercises
IOutputDevice(SessionMessageOutputDeviceData,WarningMessageOutputDeviceData,ErrorMessageOutputDeviceData), covering the 1.3.0DisplayMessagepath end to end rather than only at the serializer level. Because the run succeeds, none of these markers can come from a failure summary replay — they can only be present because they were forwarded as produced.2. Output written before the handshake completes is still surfaced —
TestProjectWithOutputBeforeHandshakeThe app writes exactly one line before
TestApplication.CreateBuilderAsync, and nothing afterwards. The SDK has to buffer it until the negotiated protocol version is known and then flush it. Producing no further output is what makes this a real guard: the flush cannot be masked by a later line draining the buffer, and the passing run rules out failure-summary replay.Validation
build.cmd -c Debugredist.ProtocolConstants.SupportedVersionsto"1.0.0"makes all 4 rows fail (the live-output rows fail withThe standard output of this test app was not observed within 60 seconds while it was still running, so it was not forwarded live.), confirming both scenarios are load-bearing rather than vacuously passing.No product code changes.