Bound post-kill WaitForExit to prevent scenario-test hangs - #354
Merged
mthalman merged 1 commit intoJul 31, 2026
Merged
Conversation
The scenario-test harness kills long-running processes (e.g. `dotnet run` on web templates) via Kill(true) followed by a parameterless WaitForExit(). With stdout/stderr redirected and read asynchronously, WaitForExit() blocks until the pipes reach EOF, not merely until the process exits. A grandchild that inherited the pipe handles and survives the tree-kill race keeps the write end open, so WaitForExit() never returns and the leg runs until the pipeline task timeout (observed as a 4h hang with no .trx). This is most frequent on the newest distro image (Fedora 43) in the source-build offline validation matrix. Bound every post-kill wait with a grace timeout so the run fails fast instead of hanging. Sites: ExecuteHelper.ExecuteProcess and DotNetSdkHelper ExecuteRunWeb/ExecuteRunUIApp. Investigation: dotnet/source-build#5624 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 367a8152-b3c0-447a-99f1-ea5d9a9d1d79
ViktorHofer
approved these changes
Jul 31, 2026
mthalman
requested changes
Jul 31, 2026
mthalman
approved these changes
Jul 31, 2026
Member
Author
|
@ViktorHofer @mthalman Thanks for your review, I don't have permission to merge this so would appreciate a merge now. |
4 tasks
Member
|
/backport to release/10.0 |
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The scenario-test harness intermittently hangs a test leg until the pipeline task timeout (observed as a 4h hang producing no
.trx). It shows up most on the newest distro image in the source-build offline validation matrix —SB_Fedora43_Offline_PreviousSourceBuiltSdk_Validation_x64— and a retry sometimes passes, so it's a race, not a deterministic failure. Tracked in dotnet/source-build#5624.Root cause
Every process-teardown path kills a long-running process (e.g.
dotnet runon a web template) withKill(true)and then calls the parameterlessWaitForExit():Because stdout/stderr are redirected and read asynchronously (
BeginOutputReadLine),WaitForExit()waits for the async readers to reach EOF on the pipes, not merely for the process to exit.dotnet runlaunches the app as a child/grandchild that inherits those pipe write handles.Kill(true)signals the process tree known at that instant, but if a descendant survives the kill for a moment (tree-walk race / reparent to pid 1), it keeps the write end open, EOF never arrives, andWaitForExit()blocks forever. Newer Fedora 43 (kernel + systemd + cgroup v2 reaping differences) makes the lingering-child race most likely, and the offlinePreviousSourceBuiltSdklegs hit the timeout→kill path most often.Fix
Bound every post-kill wait with a grace timeout (
ExecuteHelper.KillGraceMilliseconds = 30_000) so the run fails fast instead of hanging. On the already-failing/timeout path, accepting possibly-truncated output is fine. Three sites:Microsoft.DotNet.ScenarioTests.Common/ExecuteHelper.cs—ExecuteProcessMicrosoft.DotNet.ScenarioTests.SdkTemplateTests/DotNetSdkHelper.cs—ExecuteRunWebandExecuteRunUIAppMinimal, behavior-preserving for the healthy path; only the pathological "won't die / pipe held open" case changes (from infinite hang to fast failure). Built locally with
build.cmd(0 warnings, 0 errors).Investigation
Full analysis and signals: dotnet/source-build#5624
cc @akoeplinger @ViktorHofer — you're the most active maintainers here; would appreciate a review.