Harden BuildHost interactions - #84745
Conversation
|
Azure Pipelines: Successfully started running 2 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
| public void Shutdown() | ||
| { | ||
| _shutdownTokenSource.Cancel(); | ||
| _stream.Dispose(); |
There was a problem hiding this comment.
This shouldn't be needed since we're now going to unceremoniously kill the process, but it seems easier to add this than argue why it's not here, and maybe this also helps if for some reason the main process is unable to kill the process, or something else were to throw.
| if (_process.HasExited) | ||
| return; | ||
|
|
||
| if (_buildHost is not null) |
There was a problem hiding this comment.
Adding this condition to ensure that if the ConnectAsync() threw, that we aren't going to null ref and still try shutting things down cleanly.
There was a problem hiding this comment.
Pull request overview
This PR hardens MSBuild BuildHost lifecycle handling by improving cleanup paths and tightening initialization/disposal behavior so disconnected or failed-to-initialize BuildHosts don’t linger.
Changes:
- Dispose the RPC pipe stream when shutting down
RpcClient. - Ensure BuildHost processes are disposed on initialization failures and are given a short grace period to exit before being force-killed.
- Move BuildHost
Disconnectedsubscription to the point where the process is tracked in the manager.
Show a summary per file
| File | Description |
|---|---|
| src/Workspaces/MSBuild/Core/Rpc/RpcClient.cs | Disposes the underlying PipeStream during shutdown to proactively tear down the RPC transport. |
| src/Workspaces/MSBuild/Core/MSBuild/BuildHostProcessManager.cs | Improves BuildHost initialization failure cleanup and disposal/kill behavior; adjusts Disconnected event subscription timing. |
Copilot's findings
Suppressed comments (1)
src/Workspaces/MSBuild/Core/MSBuild/BuildHostProcessManager.cs:516
- DisposeAsync returns immediately when the process has already exited, which skips shutting down the RpcClient. With RpcClient.Shutdown now disposing the underlying PipeStream, this early return can leave the pipe handle undisposed in the common "process died" path.
// If the process is already exited, then we simply have nothing left to do
if (_process.HasExited)
return;
if (_buildHost is not null)
- Files reviewed: 2/2 changed files
- Comments generated: 1
dibarbet
left a comment
There was a problem hiding this comment.
@jasonmalinowski do we have anything on the buildhost side to kill itself if its parent goes away? Wonder if that is one potential cause
|
@dibarbet We do not have that today; the expectation was since we're listening to the pipe, if the app process goes away the pipe should be closed. As discussed, maybe we should be adding that, but I'd want some evidence we need that first. |
1. We had code that would call Process.Kill() if we had an exception during the shutdown process; now call it all the time if the process is still around, even if we went through a clean shutdown process. 2. Ensure that if our object is partially constructed, we'll still shut down the RpcClient and close the pipe.
We correctly killed the process if we failed to connect to the pipe, but failures during initializaton and verification it's the right MSBuild and SDK versions could have left the process running.
8c4176a to
2624f5c
Compare
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (3)
src/Workspaces/MSBuild/Core/MSBuild/BuildHostProcessManager.cs:135
- The connection-failure exception message still reads
process.ExitCodeunconditionally.Process.ExitCodethrowsInvalidOperationExceptionif the process hasn't exited yet (which can still be the case even afterDisposeAsync()attempts to kill it), potentially hiding the real connection failure. Also the message should useits(possessive) rather thanit's. Consider capturing an exit-code string guarded byHasExited(or a try/catch) and updating the wording.
await buildHostProcess.DisposeAsync().ConfigureAwait(false);
throw new Exception($"The build host was started but we were unable to connect to it's pipe. The process exited with {process.ExitCode}. Process output:{Environment.NewLine}{buildHostProcess.GetBuildHostProcessOutput()}", innerException: e);
}
src/Workspaces/MSBuild/Core/MSBuild/BuildHostProcessManager.cs:507
- Doc comment typo/grammar: "not expected to throw expectations" should be "not expected to throw exceptions" (and the first sentence currently over-promises absolute termination). Please fix the wording to match the intent.
/// <summary>
/// Shuts down the BuildHost process, ensuring the process is killed by the time this returns. This method is not expected to throw
/// expectations so callers don't have to deal with exceptions coming from broken BuildHosts.
/// </summary>
src/Workspaces/MSBuild/Core/MSBuild/BuildHostProcessManager.cs:530
DisposeAsync()returns immediately when_process.HasExitedis true. That skips_rpcClient.Shutdown(), which now disposes the pipe stream, so an unexpected process exit can leave the pipe handle undisposed in this process. Suggest removing the early return and instead only skip the remoteBuildHost.ShutdownAsynccall when the process has already exited, but still shut down the RPC client for local cleanup.
// If the process is already exited, then we simply have nothing left to do
if (_process.HasExited)
return;
if (_buildHost is not null)
{
_logger?.LogTrace("Sending a Shutdown request to the BuildHost.");
await BuildHost.ShutdownAsync(CancellationToken.None).ConfigureAwait(false);
}
if (_rpcClient is not null)
{
_rpcClient.Shutdown();
_logger?.LogTrace($"{nameof(RpcClient)} has been shut down.");
}
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
We've had some reports of BuildHosts floating around. This generally tightens up some interactions there. Commit-at-a-time recommended.
Microsoft Reviewers: Open in CodeFlow