[IBuildEngine callbacks] Stage 1: Packet infrastructure + IsRunningMultipleNodes

[JanProvaznik]

Context

TaskHost (OutOfProcTaskHostNode) doesn't support IBuildEngine callbacks - tasks that call
IsRunningMultipleNodes, BuildProjectFile, RequestCores, etc. fail with MSB5022 or
NotImplementedException. This blocks multithreaded mode (-mt) where non-thread-safe tasks
are forced to run in TaskHost.

This PR is Stage 1 of implementing callback support. It establishes the infrastructure and
implements IsRunningMultipleNodes as a proof-of-concept for the pattern.

Partial fix for #12863

Changes Made

Packet Infrastructure:

• Add packet type enum values (0x20-0x27 range) for TaskHost callbacks
• Create ITaskHostCallbackPacket interface with RequestId for request/response correlation
• Add TaskHostIsRunningMultipleNodesRequest/Response packets

OutOfProcTaskHostNode (TaskHost side):

• Add _pendingCallbackRequests dictionary mapping RequestId → TaskCompletionSource
• Add SendCallbackRequestAndWaitForResponse() - sends request, blocks task thread until response
• Add HandleCallbackResponse() - routes incoming responses to waiting TCS by RequestId
• Implement IsRunningMultipleNodes to forward query to owning worker node instead of logging error

TaskHostTask (Worker node side):

• Register handler for TaskHostIsRunningMultipleNodesRequest packets
• Forward IsRunningMultipleNodes query to real IBuildEngine and return result

Cross-Version Compatibility

New TaskHost (with callback support) may be launched by an old worker node MSBuild (without callback support). In that scenario, sending callback packets to the old worker node would cause a crash because the old worker node doesn't know how to deserialize them.

Solution: Version-gated callbacks with Traits.cs escape hatch

• PacketVersion stays at 2 throughout the staged PRs. It will be bumped to 3 only when all callback stages are complete and the feature is ready to ship.
• The TaskHost stores the worker node's PacketVersion (received during handshake in Run()).
• A CallbacksSupported property gates all callback usage:

  CallbacksSupported = _parentPacketVersion >= CallbacksMinPacketVersion  // 3
                       || Traits.Instance.EnableTaskHostCallbacks;

• When CallbacksSupported is false, callbacks return safe defaults (e.g. IsRunningMultipleNodes → false) — matching the pre-callback behavior. No packets are sent.
• For development and testing: Set MSBUILDENABLETASKHOSTCALLBACKS=1 to bypass the version check. This uses the existing Traits.cs pattern (like ForceAllTasksOutOfProcToTaskHost, EnableRarNode) which auto-refreshes per test via BuildEnvironmentState.s_runningTests.
• Integration tests set the env var to enable callbacks. A separate test verifies that callbacks are correctly disabled (returns safe defaults) when the env var is not set.

This ensures:

1. Old worker node + new TaskHost → callbacks disabled, safe defaults, no crash
2. New worker node + new TaskHost (dev/test) → env var enables callbacks before version bump
3. Final ship → PacketVersion bumped to 3, env var no longer needed, callbacks always enabled

Testing

• Added TaskHostCallbackPacket_Tests.cs - packet serialization round-trip
• Added IsRunningMultipleNodesTask.cs + integration test in TaskHostCallback_Tests.cs
• Added E2E cross-runtime test in NetTaskHost_E2E_Tests.cs
• Added test verifying callbacks disabled when env var not set (logs MSB5022 error)
• All callback and lifecycle tests pass

Notes

Callback Flow:

sequenceDiagram
    participant Task as Task Thread<br/>(TaskHost)
    participant Main as Main Thread<br/>(TaskHost)
    participant Worker as TaskHostTask<br/>(Worker Node)

    Task->>Task: BuildEngine.IsRunningMultipleNodes
    Task->>Task: Create TaskHostIsRunningMultipleNodesRequest(requestId)
    Task->>Task: Store TCS in _pendingCallbackRequests[requestId]
    Task->>Worker: SendData(request)
    Task-->>Task: Block on TCS.Task

    Worker->>Worker: HandleIsRunningMultipleNodesRequest(request)
    Worker->>Worker: result = _buildEngine.IsRunningMultipleNodes
    Worker->>Main: SendData(TaskHostIsRunningMultipleNodesResponse)

    Main->>Main: HandleCallbackResponse(response)
    Main->>Main: _pendingCallbackRequests[requestId].SetResult(response)

    Task-->>Task: TCS unblocks
    Task->>Task: return response.IsRunningMultipleNodes

Loading

Thread Model:

• Main thread runs packet dispatch loop - receives responses, routes to waiting tasks
• Task thread(s) execute task code - block on TCS when calling callbacks
• See taskhost-threading.md for detailed threading documentation

Cancellation Semantics:

• Callbacks are never interrupted by cancellation — aligned with in-process mode where IBuildEngine callbacks are direct method calls that always complete
• Cancellation is cooperative via ICancelableTask.Cancel(), which runs on the task thread after the current callback returns
• The owning worker node must send the callback response before it can exit its packet loop (causal dependency: response → task finishes → TaskHostTaskComplete → worker node exits)

Error Handling:

• Connection loss throws InvalidOperationException on pending callbacks
• New error string MSB5027: TaskHostCallbackConnectionLost

Next Stages:

• Stage 2: RequestCores/ReleaseCores
• Stage 3: BuildProjectFile/BuildProjectFilesInParallel
• Stage 4: Yield/Reacquire

[JanProvaznik]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Stage 1 of enabling IBuildEngine callback support for out-of-proc TaskHost execution by introducing request/response packet infrastructure and implementing IBuildEngine2.IsRunningMultipleNodes via a TaskHost→parent query/response roundtrip.

Changes:

• Added TaskHost callback packet types and correlation interface (ITaskHostCallbackPacket) plus query request/response packets.
• Implemented IsRunningMultipleNodes forwarding from TaskHost to parent (OutOfProcTaskHostNode ↔ TaskHostTask).
• Added unit + integration tests validating packet serialization and TaskHost callback behavior.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/Shared/TaskHostQueryResponse.cs	Adds response packet for TaskHost query callbacks.
src/Shared/TaskHostQueryRequest.cs	Adds request packet and query enum for simple callback queries.
src/Shared/ITaskHostCallbackPacket.cs	Defines request/response correlation contract via RequestId.
src/Shared/INodePacket.cs	Reserves and defines new NodePacketType values for TaskHost callbacks.
src/Samples/TaskHostCallback/TestIsRunningMultipleNodesTask.cs	Sample task exercising IBuildEngine2.IsRunningMultipleNodes in TaskHost.
src/Samples/TaskHostCallback/TestIsRunningMultipleNodes.proj	Sample project wiring task via TaskHostFactory.
src/Samples/TaskHostCallback/TaskHostCallback.csproj	Builds the sample TaskHost callback task assembly.
src/MSBuild/Resources/xlf/Strings.zh-Hant.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.zh-Hans.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.tr.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.ru.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.pt-BR.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.pl.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.ko.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.ja.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.it.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.fr.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.es.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.de.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/xlf/Strings.cs.xlf	Adds localized entry for new TaskHost connection-loss string.
src/MSBuild/Resources/Strings.resx	Adds new TaskHost connection-loss resource string.
src/MSBuild/OutOfProcTaskHostNode.cs	Implements callback request plumbing + forwards IsRunningMultipleNodes to parent.
src/MSBuild/MSBuild.csproj	Includes new Shared callback packet source files in MSBuild build.
src/Build/Microsoft.Build.csproj	Includes new Shared callback packet source files in Microsoft.Build build.
src/Build/Instance/TaskFactories/TaskHostTask.cs	Handles TaskHost query requests and replies with forwarded results.
src/Build.UnitTests/BackEnd/TaskHostQueryPacket_Tests.cs	Adds roundtrip serialization tests for query request/response packets.
src/Build.UnitTests/BackEnd/TaskHostFactory_Tests.cs	Adds integration test verifying IsRunningMultipleNodes works from TaskHost.
src/Build.UnitTests/BackEnd/IsRunningMultipleNodesTask.cs	Adds test task used by integration test to call IsRunningMultipleNodes.

[AR-May]

This looks good to me overall. My only concern is that this PR extends the same IBuildEngine callback capability not only to the sidecar task host, but also to the .NET task host. Unlike the sidecar task host, however, we cannot assume that the same MSBuild version is used - it is able to communicate cross-version.
This leads to four possible scenarios:

1. Old MSBuild node, old .NET task host
2. New MSBuild node, old .NET task host
3. Old MSBuild node, new .NET task host
4. New MSBuild node, new .NET task host

Cases 1 and 4 are fine.
In case 2, the old .NET task host would never attempt to initiate a callback request, so no callbacks would occur.
In case 3, however, the new .NET task host may attempt to initiate the callback request, which would not be understood by the old MSBuild node. We should prevent this scenario. The task host should check the negotiated communication version

msbuild/src/Shared/CommunicationsUtilities.cs

Line 179 in 5a78f0f

public byte NegotiatedPacketVersion { get; }

and fail the IBuildEngine callback attempt if the feature is not supported by the MSBuild node.

Additionally, we should ensure that all packet changes introduced by this PR and by future PRs are shipped together under a single new communication version. We do not want to support intermediate versions of the new task host communication packets.
At the same time, we should still be able to develop these changes incrementally. I suggest make the IBuildEngine callbacks disabled by default in the task host until the communication packets are finalized.