Skip to content

Benchmark the input shapes callers actually supply - #381

Merged
twcclegg merged 1 commit into
mainfrom
test/benchmark-representative-input
Aug 6, 2026
Merged

Benchmark the input shapes callers actually supply#381
twcclegg merged 1 commit into
mainfrom
test/benchmark-representative-input

Conversation

@twcclegg

@twcclegg twcclegg commented Aug 5, 2026

Copy link
Copy Markdown
Owner

An allocation probe over Parse showed the seed data is the cheapest input the library can be handed:

input B/op
+16502530000 (E164 — what the benchmark uses) 360
6502530000 national, no prefix 456
020 7031 3000 national with prefix 752
+1 650 253 0000 ext. 123 with extension 1472

PhoneNumberBenchmarkData formats everything to E164, where the national-prefix and extension regexes
both fail — and a failed match returns the shared Match.Empty for free. A successful one allocates
the Match, a GroupCollection and a Group per group. So the two things that dominate a real parse
never run in the benchmark, and any fix for them would measure as zero.

Adds ParseNationalFormat and ParseWithExtension, re-rendering the same numbers via Format.
Separate benchmarks rather than mixing them into ParseOnly: blending would give one number that
can't attribute anything, and ParseOnly stays comparable to the figure tracked across earlier PRs.

Costs two benchmark cases (~45s per tree). Lands before the optimisation so the base commit has a
counterpart to compare against.

@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.01%. Comparing base (ee09ac6) to head (ec5f50b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #381   +/-   ##
=======================================
  Coverage   98.01%   98.01%           
=======================================
  Files          39       39           
  Lines       52818    52818           
  Branches     1105     1105           
=======================================
  Hits        51770    51770           
  Misses        795      795           
  Partials      253      253           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

📊 Benchmark Results

Commit: ec5f50b · Full run · Linux ubuntu-24.04-arm

Both sides were measured on the same runner in the same job, so the numbers are
comparable. Treat sub-percent differences as noise.

PR branch

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
InputDigitPerKeystroke 1000 4.411 ms 0.0103 ms 0.0096 ms 54.6875 3.87 MB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]     : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  Job-AMQORM : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Runtime=.NET 10.0  InvocationCount=1  IterationCount=20  
LaunchCount=1  RunStrategy=ColdStart  UnrollFactor=1  
WarmupCount=1  

Method Mean Error StdDev Allocated
CreateInstance 385.0 μs 112.9 μs 130.0 μs 119.48 KB
CreateInstanceAndLoadAllRegions 6,994.3 μs 373.0 μs 429.5 μs 1619.1 KB
FirstRegionLookup 433.9 μs 125.7 μs 144.8 μs 124.54 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ExtractPossibleNumber_CleanInput 1000 21.59 μs 0.021 μs 0.019 μs - -
ExtractPossibleNumber_WithLeadingJunk 1000 38.41 μs 0.027 μs 0.022 μs 0.6714 48360 B

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
FindNumbers_Valid 100 139.6 μs 0.32 μs 0.30 μs 0.9766 69.93 KB
FindNumbers_StrictGrouping 100 313.1 μs 0.73 μs 0.69 μs 1.4648 123.2 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
GetDescriptionForNumber 1000 1,414.87 μs 2.352 μs 1.964 μs 1.9531 196.63 KB
GetDisplayCountry 1000 35.26 μs 0.070 μs 0.062 μs 0.0610 7.56 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ParseValidateAndFormatPhoneNumbers 1000 2,583.5 μs 3.15 μs 2.46 μs 7.8125 580.51 KB
ParseOnly 1000 438.2 μs 0.60 μs 0.54 μs 4.8828 350.95 KB
ParseNationalFormat 1000 839.6 μs 1.43 μs 1.34 μs 8.7891 581.22 KB
ParseWithExtension 1000 1,212.3 μs 2.33 μs 2.18 μs 21.4844 1459.2 KB
ValidateOnly 1000 764.5 μs 1.72 μs 1.53 μs - 41.11 KB
FormatOnly 1000 1,071.8 μs 2.45 μs 2.29 μs 1.9531 187.92 KB
PR base

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
InputDigitPerKeystroke 1000 4.393 ms 0.0072 ms 0.0060 ms 54.6875 3.87 MB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]     : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  Job-AMQORM : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Runtime=.NET 10.0  InvocationCount=1  IterationCount=20  
LaunchCount=1  RunStrategy=ColdStart  UnrollFactor=1  
WarmupCount=1  

Method Mean Error StdDev Allocated
CreateInstance 416.4 μs 112.4 μs 129.5 μs 119.48 KB
CreateInstanceAndLoadAllRegions 7,040.6 μs 436.4 μs 502.6 μs 1619.1 KB
FirstRegionLookup 444.5 μs 144.4 μs 166.2 μs 124.54 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ExtractPossibleNumber_CleanInput 1000 21.68 μs 0.023 μs 0.022 μs - -
ExtractPossibleNumber_WithLeadingJunk 1000 38.25 μs 0.058 μs 0.052 μs 0.6714 48360 B

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
FindNumbers_Valid 100 140.5 μs 0.23 μs 0.19 μs 0.9766 69.93 KB
FindNumbers_StrictGrouping 100 302.7 μs 0.77 μs 0.69 μs 1.4648 123.2 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
GetDescriptionForNumber 1000 1,436.30 μs 2.704 μs 2.397 μs 1.9531 196.63 KB
GetDisplayCountry 1000 22.88 μs 0.023 μs 0.022 μs 0.0916 7.56 KB

BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
Neoverse-N2, 4 physical cores
.NET SDK 10.0.302
  [Host]    : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a
  .NET 10.0 : .NET 10.0.10 (10.0.10, 10.0.1026.32716), Arm64 RyuJIT armv8.0-a

Job=.NET 10.0  Runtime=.NET 10.0  

Method PhoneNumberCount Mean Error StdDev Gen0 Allocated
ParseValidateAndFormatPhoneNumbers 1000 2,572.1 μs 4.79 μs 4.48 μs 7.8125 580.51 KB
ParseOnly 1000 435.4 μs 0.59 μs 0.52 μs 4.8828 350.95 KB
ValidateOnly 1000 774.0 μs 1.43 μs 1.20 μs - 41.11 KB
FormatOnly 1000 1,075.5 μs 3.99 μs 3.73 μs 1.9531 187.92 KB

@twcclegg
twcclegg merged commit af40c92 into main Aug 6, 2026
7 checks passed
@twcclegg
twcclegg deleted the test/benchmark-representative-input branch August 6, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant