Skip to content

ci: skip dotnet's first-time-experience setup in run_performance_tests - #411

Merged
twcclegg merged 1 commit into
mainfrom
ci/skip-dotnet-first-time-experience-in-perf-tests
Aug 24, 2026
Merged

ci: skip dotnet's first-time-experience setup in run_performance_tests#411
twcclegg merged 1 commit into
mainfrom
ci/skip-dotnet-first-time-experience-in-perf-tests

Conversation

@twcclegg

Copy link
Copy Markdown
Owner

Summary

run_performance_tests.yml's "Build both benchmark trees" step runs two dotnet build invocations concurrently via & — the only workflow that does, since everywhere else an earlier sequential dotnet restore/build step already completes the .NET SDK's one-time first-use setup (NuGet migrations, telemetry sentinel) before anything runs in parallel. Here, both concurrent invocations can race to perform that setup at once, both trying to create the same NuGet-Migrations mutex/directory, and one loses with:

System.IO.IOException: The system cannot open the device or file specified. : 'NuGet-Migrations'.
One or more system calls failed: mkdir("/tmp/.dotnet/shm/session...", ...) == -1; errno == EEXIST;

Seen failing PR #409's run_performance_tests check — unrelated to that PR's actual diff, purely this race.

DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true (plus DOTNET_NOLOGO: true for consistency) removes the race by skipping the setup entirely rather than reordering around it.

Test plan

  • Validated workflow YAML parses correctly
  • Root cause confirmed by log inspection: no other workflow parallelizes dotnet build invocations, and none currently set these env vars

Build both benchmark trees runs two dotnet build invocations concurrently
via & - the only workflow that does, since everywhere else an earlier
sequential dotnet restore/build step already completes the SDK's one-time
setup (NuGet migrations, telemetry sentinel) before anything runs in
parallel. Here the two concurrent invocations can race to perform that
setup at once, both trying to create the same NuGet-Migrations mutex/
directory, and lose with "The system cannot open the device or file
specified" / EEXIST - seen failing PR #409's run_performance_tests job.

DOTNET_SKIP_FIRST_TIME_EXPERIENCE removes the race by skipping the setup
entirely rather than reordering around it.
@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.26%. Comparing base (a40ff86) to head (9567bac).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #411   +/-   ##
=======================================
  Coverage   77.26%   77.26%           
=======================================
  Files          39       39           
  Lines        4548     4548           
  Branches     1129     1129           
=======================================
  Hits         3514     3514           
  Misses        783      783           
  Partials      251      251           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown

📊 Benchmark Results

Commit: 9567bac · Full run · Linux ubuntu-24.04-arm

Both sides were measured on the same runner in the same job, so the numbers are
comparable. Treat sub-percent differences as noise.

PR branch

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
InputDigitPerKeystroke 1000 4.658 ms 0.0528 ms 0.0494 ms 54.6875 3.87 MB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]     : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  Job-AMQORM : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Runtime=.NET 10.0  InvocationCount=1  IterationCount=20  
LaunchCount=1  RunStrategy=ColdStart  UnrollFactor=1  
WarmupCount=1  

Method Mean Error StdDev Allocated
CreateInstance 436.5 μs 122.2 μs 140.7 μs 119.48 KB
CreateInstanceAndLoadAllRegions 7,157.0 μs 418.1 μs 481.4 μs 1620.34 KB
FirstRegionLookup 470.7 μs 152.5 μs 175.6 μs 124.54 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ExtractPossibleNumber_CleanInput 1000 20.88 μs 0.015 μs 0.014 μs - -
ExtractPossibleNumber_WithLeadingJunk 1000 38.32 μs 0.058 μs 0.054 μs 0.6714 48360 B

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
FindNumbers_Valid 100 146.0 μs 0.22 μs 0.20 μs 0.9766 70.71 KB
FindNumbers_StrictGrouping 100 314.6 μs 1.26 μs 0.98 μs 1.4648 123.99 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
GetDescriptionForNumber 1000 1,433.01 μs 6.952 μs 5.805 μs 1.9531 196.59 KB
GetDisplayCountry 1000 18.72 μs 0.020 μs 0.017 μs 0.0916 7.56 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ParseValidateAndFormatPhoneNumbers 1000 2,627.7 μs 17.54 μs 15.55 μs 7.8125 577.71 KB
ParseOnly 1000 445.0 μs 0.70 μs 0.62 μs 4.8828 348.13 KB
ParseNationalFormat 1000 774.9 μs 1.58 μs 1.48 μs 5.8594 431.36 KB
ParseWithExtension 1000 1,119.7 μs 2.02 μs 1.79 μs 13.6719 957.38 KB
ValidateOnly 1000 761.5 μs 1.83 μs 1.71 μs - 41.11 KB
FormatOnly 1000 1,103.3 μs 2.21 μs 1.84 μs 1.9531 187.92 KB
PR base

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
InputDigitPerKeystroke 1000 4.902 ms 0.0939 ms 0.0878 ms 54.6875 3.87 MB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]     : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  Job-AMQORM : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Runtime=.NET 10.0  InvocationCount=1  IterationCount=20  
LaunchCount=1  RunStrategy=ColdStart  UnrollFactor=1  
WarmupCount=1  

Method Mean Error StdDev Allocated
CreateInstance 414.2 μs 128.0 μs 147.4 μs 119.48 KB
CreateInstanceAndLoadAllRegions 7,144.1 μs 425.7 μs 490.2 μs 1620.34 KB
FirstRegionLookup 486.8 μs 170.4 μs 196.3 μs 124.54 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ExtractPossibleNumber_CleanInput 1000 20.94 μs 0.020 μs 0.019 μs - -
ExtractPossibleNumber_WithLeadingJunk 1000 38.66 μs 0.035 μs 0.029 μs 0.6714 48360 B

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
FindNumbers_Valid 100 144.0 μs 0.17 μs 0.15 μs 0.9766 70.71 KB
FindNumbers_StrictGrouping 100 314.9 μs 0.84 μs 0.79 μs 1.4648 123.99 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
GetDescriptionForNumber 1000 1,505.18 μs 4.936 μs 4.617 μs 1.9531 196.59 KB
GetDisplayCountry 1000 18.54 μs 0.042 μs 0.040 μs 0.0916 7.56 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.400
  [Host]    : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.11 (10.0.11, 10.0.1126.37416), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ParseValidateAndFormatPhoneNumbers 1000 2,653.4 μs 8.88 μs 7.42 μs 7.8125 577.71 KB
ParseOnly 1000 439.4 μs 0.50 μs 0.45 μs 4.8828 348.13 KB
ParseNationalFormat 1000 783.3 μs 2.20 μs 2.06 μs 5.8594 431.36 KB
ParseWithExtension 1000 1,122.8 μs 1.37 μs 1.21 μs 13.6719 957.38 KB
ValidateOnly 1000 769.0 μs 1.91 μs 1.70 μs - 41.11 KB
FormatOnly 1000 1,110.1 μs 4.22 μs 3.74 μs 1.9531 187.92 KB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant