Investigate stability of PGO updates

[AndyAyersMS]

We are seeing fairly frequent microbenchmark performance shifts with the managed PGO updates. We need to root cause this and figure out how to address them.

Possible causes:

• non-deterministic training scenarios
• non-deterministic collection behavior (even for deterministic scenarios)
• issues in exfiltrating the profile data into the trace records
• issues in converting the trace records to the .mibc format
• issues merging multiple runs together into a consolidated .mibc file
• issues incorporating the .mibc profile data into assemblies (see eg ...)
• schema version skew leading to the jit dropping profile data
• other issues causing the jit to drop profile data
• jit optimizations overly sensitive to small changes in profile data

For example, the last few fluctuations here are correlated with PGO updates:

See also

• [Perf] Changes at 5/6/2021 7:25:59 PM DrewScoggins/performance-2#5747
• [Perf] Changes at 5/6/2021 11:39:12 PM DrewScoggins/performance-2#5742
• [Perf] Changes at 5/6/2021 7:25:59 PM DrewScoggins/performance-2#5741
• [Perf] Changes at 5/6/2021 2:48:40 PM DrewScoggins/performance-2#5783

First step is perhaps to create a .mibc comparison tool (or mode in in the pgo tool) to try and see how much variation we're seeing from one update to the next. Given two .mibc it would first match up methods and report how many methods have PGO data in one collection but not the other; then for cases where both methods have PGO, it could detect when the schemas have changed, and where the schemas agree, it could run a similarity analysis on the count and class profile data to see if what was measured seems to have changed in any significant way.

category:cq
theme:profile-feedback

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AndyAyersMS]

Marking as codegen for now, though the issues may lie elsewhere.

[AndyAyersMS]

For System.Collections.ContainFalse<String>.List, I get these results running locally, where the commits fall in the three ranges below from the lab results:

BenchmarkDotNet=v0.12.1.1528-nightly, OS=Windows 10.0.19043
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.100-preview.5.21227.8
  [Host]     : .NET 6.0.0 (6.0.21.22701), X64 RyuJIT
  Job-RIQHCA : .NET 6.0.0 (42.42.42.42424), X64 RyuJIT

Commit	Method	Size	Mean	Error	StdDev	Median	Min	Max
6aca7af	List	512	963.4 us	10.30 us	9.64 us	965.6 us	944.4 us	975.2 us
e3db83f	List	512	970.1 us	9.75 us	8.64 us	968.7 us	958.7 us	986.9 us
60e6244	List	512	966.7 us	8.50 us	7.53 us	965.8 us	954.3 us	980.9 us

[AndyAyersMS]

For Span.IndexerBench.KnownSizeArray, no local repro either.

Commit	Method	length	Mean	Error	StdDev	Median	Min	Max
6aca7af	KnownSizeArray	1024	478.5 ns	4.95 ns	4.39 ns	478.9 ns	467.4 ns	485.6 ns
e3db83f	KnownSizeArray	1024	478.9 ns	5.07 ns	4.75 ns	479.8 ns	470.9 ns	485.4 ns
60e6244	KnownSizeArray	1024	483.6 ns	4.12 ns	3.85 ns	484.2 ns	476.8 ns	490.2 ns

[AndyAyersMS]

@jakobbotsch has developed some .mibc comparsion tech and overall the managed profile data produced by the optimization repo looks fairly stable from day to day.

[AndyAyersMS]

Given the relative stability of the PGO collection I'm going to close this for now.

The perf fluctuations coming from PGO updates are proving tough to pin down, but the root cause doesn't appear to be managed PGO instability.

[AndyAyersMS]

I'm going to re-open this. We continue to see a steady rate of perf fluctuations from PGO. These have become more obvious during the .NET 6 endgame because we still see daily PGO updates in main and there's not much else going on.

A few possible areas for deeper analysis:

• PGO collection is inherently unstable, since we are measuring low-level details of non-deterministic applications. So some basic level of instability needs to be tolerated.
• PGO collection relies on randomization (for class profiles)
• PGO collection spans some 40 odd processes and the resulting merge algorithm may need some adjusting (see eg PGO: class profile details we need to get right #48549 (comment) )
• PGO coverage in those processes needs to be improved in some areas (eg locales, [Perf] Regressions in System.Text.Perf_Utf8Encoding for Greek and Cyrillic #52313)
• The jit will likely make important choices (eg block ordering) based on relatively thin evidence ([Perf] Regressions in System.Collections.TryGetValueFalse<String, String> #51258).
• The jit is careless with updating and maintaining profile data
• The jit doesn't handle mixtures of IR with PGO and without PGO well (say from inlining). See eg [Perf] Regressions in IndexOf #54165.
• The jit likely has poor heuristics in many areas, so better information in does not lead to better code out

As far as tolerating the inherent instability, @tannergooding has suggested we might blend PGO data collected on successive days as a way of stabilizing updates; this seems like it has promise.

[AndyAyersMS]

As far as I know this is not causing problems, though we do see perf swings in a number of tests from PGO updates.

Going to move to future.