Skip to content

Serialize concurrent .NET tool operations with file locks - #51834

Open
baronfel with Copilot wants to merge 15 commits into
release/11.0.1xxfrom
copilot/fix-concurrent-tool-installation
Open

Serialize concurrent .NET tool operations with file locks#51834
baronfel with Copilot wants to merge 15 commits into
release/11.0.1xxfrom
copilot/fix-concurrent-tool-installation

Conversation

Copilot AI commented Nov 20, 2025

Copy link
Copy Markdown
Contributor

Concurrent tool operations can race while downloading packages, mutating the global tool store, or creating shared shims. This change serializes those operations with persistent file locks opened using FileShare.None.

Changes

  • Use package/version locks for local package extraction and a store-wide lock for global/tool-path install, update, and uninstall operations.
  • Keep lock files on disk so stale files are harmless; ownership is represented only by the open exclusive handle.
  • Retain package and store locks through ambient transaction completion so rollback cannot delete or restore over a subsequent operation.
  • Normalize package identities and lock paths across casing and DOTNET_CLI_HOME differences.
  • Make tool install, restore, update, execute, and uninstall paths asynchronous and propagate cancellation through NuGet lookup, download, extraction, and lock waits.
  • Preserve --disable-parallel behavior for tool restore and revalidate update --all entries under the store lock.
  • Add concurrency, cancellation, rollback, stale-lock, RID-package, shared-shim, and update/uninstall regression coverage.

When contention is detected, the CLI reports that another installation is in progress and waits up to the existing timeout. Users can cancel the wait with Ctrl+C.

Fixes #51831

Copilot AI changed the title [WIP] Fix concurrent tool installation issue with named mutex Fix concurrent tool installation race conditions with named mutex Nov 20, 2025
Copilot AI requested a review from baronfel November 20, 2025 21:17
Comment thread src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Outdated
Copilot AI requested a review from baronfel November 20, 2025 21:51
@baronfel
baronfel marked this pull request as ready for review November 20, 2025 23:47
Copilot AI review requested due to automatic review settings November 20, 2025 23:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request implements concurrent tool installation protection using named mutexes to prevent "file being used by another process" errors when multiple dotnet CLI processes attempt to install the same .NET tool simultaneously.

Key changes:

  • Adds named mutex synchronization (tool-install-{packageId}-{packageVersion}) to serialize concurrent installations per package/version combination
  • Implements two-stage mutex acquisition with user notification for better UX
  • Adds comprehensive test coverage for concurrent installation scenarios

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

File Description
src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Core implementation: wraps DownloadTool method with named mutex, includes timeout handling and mutex name generation
src/Cli/dotnet/CliStrings.resx Adds two new error messages: ToolInstallationTimeout and ToolInstallationWaiting for user communication
src/Cli/dotnet/xlf/*.xlf Adds localization entries for the new error messages across 13 language files (all marked as state="new")
test/Microsoft.DotNet.PackageInstall.Tests/ToolPackageDownloaderTests.cs Adds GivenConcurrentInstallationsTheyDoNotConflict test with concurrent Task.Run executions; includes minor formatting fixes (alphabetized using statements, whitespace cleanup)

Comment thread src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs
Comment thread src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Outdated
Comment thread src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Outdated
Comment thread src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Outdated
Comment thread src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Due to lack of recent activity, this PR has been labeled as 'Stale'. It will be closed if no further activity occurs within 7 more days. Any new comment will remove the label.

@github-actions github-actions Bot added the stale label Apr 23, 2026
@github-actions github-actions Bot closed this Apr 30, 2026
@baronfel baronfel reopened this Aug 26, 2026
@baronfel
baronfel requested a review from a team as a code owner August 26, 2026 21:09
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI and others added 3 commits August 26, 2026 16:11
Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>
- Add 50ms initial WaitOne to quickly check if mutex is available
- Display informative message when waiting for another installation
- Tell users they can press Ctrl+C to cancel
- Maintains 5-minute total timeout for full wait

Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>
@baronfel
baronfel changed the base branch from main to release/11.0.1xx August 26, 2026 21:12
@baronfel
baronfel force-pushed the copilot/fix-concurrent-tool-installation branch from e779d23 to 137d0f7 Compare August 26, 2026 21:12
@baronfel
baronfel requested review from a team as code owners August 26, 2026 21:12
baronfel and others added 9 commits August 26, 2026 16:17
Release the mutex only when the current thread acquired it so timeout and acquisition failures are not masked.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Treat AbandonedMutexException as successful acquisition so a prior installer crash does not block the next installation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Hash the normalized install root, package ID, and version into a bounded mutex name so unrelated agents and directories do not contend.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Flow System.CommandLine cancellation through install, update, restore, execute, and dnx so Ctrl+C interrupts a contended mutex wait.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Keep the per-package mutex through the staged package move, package validation, and rollback. Recheck the destination after acquiring the mutex so a concurrent installer reports the existing-package conflict instead of racing the final move.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Preserve the legacy downloader overload while adding a cancellation-aware overload, and add focused coverage for contention, timeout, cancellation, abandonment, scoped mutex identities, and complete global installation serialization.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Move invocation cancellation from command constructors to Execute methods and propagate it through install, update, restore, and tool execution paths.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Use one InstallPackage overload with an optional trailing cancellation token.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Return Task<int> from cancellation-aware Execute overloads so command parser actions can return command tasks directly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@github-actions

This comment has been minimized.

Replace thread-affine named mutexes with persistent FileShare.None locks, keep locks through ambient transaction completion, and propagate cancellation through asynchronous install, update, restore, execute, and uninstall paths.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@baronfel baronfel changed the title Fix concurrent tool installation race conditions with named mutex Serialize concurrent .NET tool operations with file locks Aug 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Build Failure Analysis — MSTEST0049 errors (all 10 legs)

All 10 build legs fail with MSTEST0049 build errors in two test projects:

  • test/dotnet.Tests/dotnet.Tests.csproj — 11 errors
  • test/Microsoft.DotNet.PackageInstall.Tests/Microsoft.DotNet.PackageInstall.Tests.csproj — 39 errors

Root Cause

This PR changed IToolPackageDownloader to expose async methods (InstallPackageAsync, GetNuGetVersionAsync, TryGetDownloadedToolAsync) each with a CancellationToken cancellationToken = default parameter, and added sync extension helpers (InstallPackage, GetNuGetVersion) that also expose CancellationToken. The [MSTEST0049]((learn.microsoft.com/redacted) analyzer fires as a build error (because both MSTest.Sdk-based projects treat analyzer warnings as errors) whenever a test calls one of these methods through an Action lambda or directly without passing TestContext.CancellationToken.

Files modified by this PR that have errors

test/dotnet.Tests/CommandTests/Tool/Uninstall/ToolUninstallGlobalOrToolPathCommandTests.cs — 5 errors at lines 51, 87, 194, 234, 259

test/Microsoft.DotNet.PackageInstall.Tests/ToolPackageDownloaderTests.cs — 25 errors at lines 61, 88, 113, 155, 178, 203, 221, 243, 266, 299, 327, 352, 385, 420, 430, 479, 529, 541, 547, 571, 583, 584, 592, 621, 629

Other files with pre-existing errors (NOT modified by this PR, but also failing)

The following test files were last touched by PR #54646 ("Resolve analyzers from the analyzers group in project.assets.json"), which likely activated the MSTEST0049 analyzer. They have pre-existing violations that were already present on the base branch but now become build errors:

  • test/dotnet.Tests/CommandTests/Tool/Uninstall/ToolUninstallCommandTests.cs — 4 errors (lines 32, 47, 62, 76)
  • test/dotnet.Tests/CommandTests/Tool/Uninstall/ToolUninstallLocalCommandTests.cs — 1 error (line 123)
  • test/dotnet.Tests/CommandTests/Tool/Install/ToolInstallGlobalOrToolPathCommandTests.cs — 1 error (line 820)
  • test/Microsoft.DotNet.PackageInstall.Tests/NuGetPackageInstallerExtractTests.cs — 2 errors
  • test/Microsoft.DotNet.PackageInstall.Tests/NuGetPackageInstallerTests.cs — 3 errors
  • test/Microsoft.DotNet.PackageInstall.Tests/ToolPackageInstallerNugetCacheTests.cs — 2 errors
  • test/Microsoft.DotNet.PackageInstall.Tests/ToolPackageUninstallerTests.cs — 1 error

Note: If the base branch build is also failing due to those pre-existing violations, this PR cannot land until they are resolved too. In that case, this PR author only needs to fix the violations in files they modified; the pre-existing violations are separate.

Fix

For every call site flagged by MSTEST0049, pass cancellationToken: TestContext.CancellationToken (where the test class has access to TestContext) to the optional cancellationToken parameter of InstallPackage, GetNuGetVersion, etc.

Example pattern — before:

var package = downloader.InstallPackage(
    new PackageLocation(nugetConfig: testDir.WithFile("NuGet.config")),
    packageId: TestPackageId,
    verbosity: TestVerbosity,
    versionRange: VersionRange.Parse(TestPackageVersion),
    targetFramework: _testTargetframework,
    isGlobalTool: true,
    verifySignatures: false);

After:

var package = downloader.InstallPackage(
    new PackageLocation(nugetConfig: testDir.WithFile("NuGet.config")),
    packageId: TestPackageId,
    verbosity: TestVerbosity,
    versionRange: VersionRange.Parse(TestPackageVersion),
    targetFramework: _testTargetframework,
    isGlobalTool: true,
    verifySignatures: false,
    cancellationToken: TestContext.CancellationToken);

For Action lambdas used with .Should().Throw<>(), also pass TestContext.CancellationToken:

// Before
Action secondCall = () => downloader.InstallPackage(..., verifySignatures: false);
secondCall.Should().Throw<ToolPackageException>()...;

// After
Action secondCall = () => downloader.InstallPackage(..., verifySignatures: false,
    cancellationToken: TestContext.CancellationToken);
secondCall.Should().Throw<ToolPackageException>()...;

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · 294.5 AIC · ⌖ 7.94 AIC · ⊞ 5.3K · [◷]( · )

@github-actions github-actions Bot removed the stale label Aug 27, 2026
Pass MSTest's cancellation token to the new cancellable tool APIs so analyzer enforcement does not fail CI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent Tool Installation can fail

4 participants