Add support for dumping and using precise debug info #61735
Conversation
* In the JIT, add support for dumping the precise debug info out through an environment variable `DOTNET_JitDumpPreciseDebugInfoFile` in a simple JSON format. This is a stopgap until we expose the extra information through ETW events. * In dotnet-pgo, add an argument --precise-debug-info-file which can point to the file produced by the JIT. When used, dotnet-pgo will get native<->IL mappings from this file instead of through ETW events. * In dotnet-pgo, add support for attributing samples to inlinees when that information is present. This changes the attribution process a bit: previously, we would group all LBR data/samples and then construct the profile from all the data. We now do it in a more streaming way where there is a SampleCorrelator that can handle individual LBR records and individual samples. * In dotnet-pgo, add an argument --dump-worst-overlap-graphs-to which can be used in the compare-mibc command to dump out a .dot file containing the flow graph of the methods with the worst overlap measures, and showing the relative weight count on each basic block and edge for the two profiles being compared. This is particular useful to find out where we are producing incorrect debug mappings, by comparing spgo.mibc and instrumented.mibc files.
|
I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label. |
|
@AndyAyersMS @davidwrighton Can you please take a look? No need to review the SPGO parts in detail yet since it's more or less at a prototype stage. |
src/coreclr/jit/codegencommon.cpp
Outdated
| while (InterlockedCompareExchange(&s_flag, 1, 0) != 0) | ||
| System_YieldProcessor(); |
There was a problem hiding this comment.
Do we have a better lock primitive in the JIT I can use here?
There was a problem hiding this comment.
There is CritSecHolder, if that would be suitable here.
There was a problem hiding this comment.
That looks better, thanks.
|
The dumped flow graphs look like this: This is very helpful to see that clearly the attribution process is not working well for in this particular function. |
jakobbotsch
added
the
area-CodeGen-coreclr
|
Tagging subscribers to this area: @JulieLeeMSFT Issue Details
|
| public long TotalEdgeCount1 => BasicBlocks.Sum(bb => bb.Edges.Sum(e => e.Value.Count1)); | ||
| public long TotalEdgeCount2 => BasicBlocks.Sum(bb => bb.Edges.Sum(e => e.Value.Count2)); | ||
|
|
||
| public double ComputeBlockOverlap() |
There was a problem hiding this comment.
You might also look into the Spearman rank correlation as a measure of similarity.
| if (JitConfig.JitDumpPreciseDebugInfoFile() == nullptr) | ||
| return; | ||
|
|
||
| static CritSecObject s_critSect; |
There was a problem hiding this comment.
Generally we don't want the jit to take locks but using one during bring-up like this is ok.
| /// Given some IL offset samples into a method, construct a profile. | ||
| /// </summary> | ||
| public static SampleProfile Create(MethodIL il, FlowGraph fg, IEnumerable<int> ilOffsetSamples) | ||
| public void SmootheFlow() |
There was a problem hiding this comment.
Nit ..?
| public void SmootheFlow() | |
| public void SmoothFlow() |
There was a problem hiding this comment.
Today I learned that 'smooth' can also be used as a verb. :-)
|
Gonna merge this as I have some follow-up work that builds on it. @davidwrighton let me know if you have any feedback and I'll address it separately. |
ghost
locked as resolved and limited conversation to collaborators
In the JIT, add support for dumping the precise debug info out through
an environment variable
DOTNET_JitDumpPreciseDebugInfoFilein asimple JSON format. This is a stopgap until we expose the extra
information through ETW events.
In dotnet-pgo, add an argument --precise-debug-info-file which can
point to the file produced by the JIT. When used, dotnet-pgo will get
native<->IL mappings from this file instead of through ETW events.
In dotnet-pgo, add support for attributing samples to inlinees when
that information is present. This changes the attribution process a
bit: previously, we would group all LBR data/samples and then
construct the profile from all the data. We now do it in a more
streaming way where there is a SampleCorrelator that can handle
individual LBR records and individual samples.
In dotnet-pgo, add an argument --dump-worst-overlap-graphs-to which
can be used in the compare-mibc command to dump out a .dot file
containing the flow graph of the methods with the worst overlap
measures, and showing the relative weight count on each basic block
and edge for the two profiles being compared. This is particular
useful to find out where we are producing incorrect debug mappings, by
comparing spgo.mibc and instrumented.mibc files.