perf(test): add first-use-per-region cold-start benchmark - #438
Merged
Conversation
Every existing benchmark here either repeats one region (cheap after the first call) or warms its whole diverse region set in GlobalSetup before the timed run starts (PhoneNumberWorkflowBenchmark's seed-data generation calls GetExampleNumberForType/IsValidNumber/Format against every supported region). Neither shape can see the cost of a region's genuinely first use in the process, which is exactly where two real regressions lived: the 2017 RegexOptions.Compiled + undersized RegexCache issue (~115x, fixed by PR #161) and the 2026 PR #325 Compiled-regex change (~100x cold-start cost per new region, not caught by its own benchmark because that benchmark's setup already pre-warms every region it measures). Add ColdStartBenchmark.FirstUseValidateAndFormat: fresh PhoneNumberUtil per iteration, a fixed 20-region list GlobalSetup never touches, one previously-unseen region's full Parse+IsValidNumber+Format per invocation. Locally: 19.6ms median vs FirstRegionLookup's 300us (metadata-load-only), cleanly separating the regex-compile cost from the metadata-load cost this class already measured. No wiring changes needed - run_performance_tests.yml already runs `--filter "*"` for both branch and base and diffs every case via lib/compare-benchmarks.js's Welch's-t-test + 20%-floor comparison, so the new case is covered automatically. Documented the two-regression history and the "why" for this specific benchmark shape in README.md so a future benchmark addition doesn't accidentally drop the property that makes this one work.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #438 +/- ##
=======================================
Coverage 87.26% 87.26%
=======================================
Files 41 41
Lines 3831 3831
Branches 978 978
=======================================
Hits 3343 3343
Misses 284 284
Partials 204 204 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
FirstUseValidateAndFormat only covered the Parse/IsValidNumber/Format path. AsYouTypeFormatter and PhoneNumberMatcher share the same PhoneRegex pattern cache and are exposed to the identical class of regression; PhoneNumberOfflineGeocoder has an analogous lazy-load blind spot in its own benchmark class. Add FirstUseAsYouType, FirstUseFindNumbers, and FirstUseGeocode, each with its own disjoint 15-region pool so no benchmark in the process pre-warms another's regions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ColdStartBenchmark.FirstUse*benchmarks, each doing a real end-to-end call against a fresh instance (or, forFirstUseAsYouType, a fresh formatter off the shared warmPhoneNumberUtil) targeting one region from a fixed, disjoint pool thatGlobalSetupnever pre-warms — one previously-unseen region per invocation:FirstUseValidateAndFormat—Parse+IsValidNumber+Format(20 regions)FirstUseAsYouType—AsYouTypeFormatter.InputDigit, keystroke by keystroke (15 regions)FirstUseFindNumbers—PhoneNumberUtil.FindNumbers(15 regions)FirstUseGeocode—PhoneNumberOfflineGeocoder.GetDescriptionForNumber, via the internal constructor so each invocation gets an un-warmed prefix-map cache (15 regions)GlobalSetuptouches — necessary because every benchmark class in this project runs inside one process, andPhoneRegex's pattern cache (and the geocoder's prefix-map cache) are process-wide, not per-instance.ColdStartBenchmarkbenchmarks (metadata-loading only) and the other classes' steady-state benchmarks (diverse regions, but pre-warmed in setup) both stay exactly as they are; they're good and valid for what they measure.README.md, including the two real regressions (#161's fix for the 2017 issue, and #325's 2026 cold-start tradeoff) that this exact gap let ship unnoticed, plus the disjoint-pool convention so a future benchmark addition doesn't accidentally re-warm a region anotherFirstUse*benchmark depends on staying cold.Why this specific gap
Every existing benchmark here either repeats one region (cheap after the first call) or warms its whole diverse region set in
GlobalSetupbefore the timed run starts — e.g.PhoneNumberWorkflowBenchmark's seed-data generation callsGetExampleNumberForType/IsValidNumber/Formatagainst every supported region while building its dataset, andAsYouTypeFormatterBenchmark/PhoneNumberMatcherBenchmark/PhoneNumberOfflineGeocoderBenchmarkdo the equivalent for their own subsystems. None of these shapes can see the cost of a region's genuinely first use in the process.PhoneRegex's pattern cache (PhoneRegex.cs) is astatic ConcurrentDictionary, process-wide — a fresh instance doesn't reset it, only a never-before-touched region does. The geocoder's prefix-map cache (PrefixFileReader.cs) is architecturally different (lazy-loaded maps, no regex) but has the identical structural blind spot in its own benchmark class, so it's included for symmetry/completeness across the library's region-keyed subsystems.Locally, all four
FirstUse*benchmarks show multi-millisecond medians versusFirstRegionLookup's (metadata-load-only) ~300us — cleanly separating the first-touch regex-compile/lazy-load cost this class was missing from the metadata-load cost it already measured.CI wiring — already fully automatic, no changes needed
run_performance_tests.ymlruns--filter "*"for both the branch and the PR base,lib/compare-benchmarks.jsdiffs every matching case (Welch's t-test + a 20% relative-delta floor, chosen well above measured single-launch noise), andlib/fail-on-benchmark-regression.jsexits 1 — failing the workflow — ifsignificant-changes.jsonhas anyregressionsentries, regardless of which specific case regressed.post_performance_test_comment.yml(a separateworkflow_run-triggered job) posts/updates a "📊 Statistically significant benchmark improvement" comment with a markdown table wheneversignificant-changes.jsonhas anyimprovementsentries, and removes the comment if a later push has none.FirstUse*cases are picked up automatically.Test plan
dotnet build csharp— whole solution builds cleandotnet test csharp/PhoneNumbers.slnx -p:TargetFrameworks=net10.0— 451/451 pass, unaffected (PerformanceTest-only change)dotnet run -c Release --framework net10.0 -- --filter "*ColdStartBenchmark*"locally — all seven benchmarks run cleanly, no exceptions/NA, eachFirstUse*benchmark shows the expected cold-vs-warm separation