Investigating analyzer impact on build times

[peteroupc]

StyleCop Classic used to write information about files it analyzed, including analysis results, in an XML cache called StyleCop.Cache, to speed the time it takes to analyze a project if some of the source code files are unchanged in the meantime. Although StyleCop Analyzers also has caches, they are limited to storing using aliases and copyright texts, but nothing like a results cache like StyleCop Classic used to have.

In my experience, running the StyleCop analyzer takes a considerable part of a project's build time even if it ultimately finds few or no violations, and I feel that having a results cache (like StyleCop.Cache) will speed up this time because StyleCop would then focus analysis on only those files that have changed, which is often not many files.

[sharwell]

@peteroupc StyleCop Analyzers is not StyleCop Classic. If you find an indication of a performance problem, please file an issue and we will work to resolve it directly.

[peteroupc]

I can't speak of the slow performance of StyleCop Analyzers except in general terms. I know that StyleCop Analyzers is more than doubling build time of my projects but I don't know where to start in finding a specific "performance problem" of the kind you have in mind.

[sharwell]

What was the build time prior to StyleCop Analyzers, and the build time after?

Do you have any other analyzers installed in your project?

[peteroupc]

For one project of mine, the build time (msbuild /t:rebuild) was—

• about 1:06 without StyleCop Analyzers, and
• about 2:54 with StyleCop Analyzers,

resulting in almost triple the build time. This project used no other analyzers than StyleCop Analyzers.

[sharwell]

@peteroupc The last time I measured it, the performance cost of StyleCop Analyzers was about 90% in the analyzer driver (enabled whenever any analyzer is installed and enabled) and about 10% StyleCop Analyzers itself. This project only has control over the performance of the latter. The easiest way to see if this is impacting you is to install another analyzer, e.g. FXCop Analyzers, and then re-run your comparison between StyleCop Analyzers installed or not. In this scenario, I would expect the difference between the two cases to be much less than 1:48.

The other thing that can impact performance is the number of diagnostics reported by the analyzers. This is the same as the number of diagnostics you see if the project has no suppressions. StyleCop Analyzers is optimized for cases where no diagnostics are reported. A few diagnostics wouldn't make much difference, but if are seeing and/or suppressing thousands of diagnostics, it would have a noticeable impact.

[peteroupc]

The build time for the same project is—

• about 1:39 with FxCop analyzers version 2.6.4, and
• about 3:22 with FxCop analyzers version 2.6.4 and StyleCop Analyzers.

Also, several rules were disabled in the project in question; enabling them raised the build time to about 4:01 (with just StyleCop Analyzers) and added almost 5000 additional warnings (mostly SA1500 and SA1513); though this may be due to overhead from displaying those warnings.

[sharwell]

Can you try adding /p:ReportAnalyzer=true to your MSBuild command line to see execution times by analyzer?

[sharwell]

@peteroupc If the results from ReportAnalyzer don't show in your output, you can add /bl:Build.binlog to your MSBuild command line as well. This will create an output file Build.binlog that you can open with the MSBuild Structured Log Viewer. This output format contains the information you would normally see with diagnostic level output, but is much faster than /v:d and the log viewer contains search functionality.

When you open the log in the viewer, you'll want to search for the following:

"Total analyzer execution time"

Then you can click on a line on the left pane to get details.