Feature Request: Extract Analyzer diagnostics post build event

[Cyassin]

@jinujoseph @mavasani - as discussed.

Our organisation currently uses our own built Roslyn analyzers for highlighting issues in the IDE but also as part of our CI/CD pipeline. Before we turn analyzers from warning to error we want to be able to extract stats and create our own burn-downs of these results in the build pipeline to ensure we don't stop rolling changes when a new analyzer is first added to the build.

We have looked into using Analyzer hosts which work well at doing this, but requires us to duplicate our publish builds with our analyzer host build which duplicates effort in the build pipeline. Our rules don't add much overhead to analysis times in the build pipeline but by duplicating compiles of some large projects does blow out full pipeline times.

We have also tried utilizing the context.RegisterCompilationEndAction() event on an individual analyzer with our own static class object inherited by all analyzers that we write the diagnostics to. This also works but makes unit testing extremely difficult with the shared context and isn't a very neat solution.

What would be ideal is a feature so that we could write an event or override on the whole of compilation so that we gather the post build information from the analyzers and do something with it. I vision something similar to the .NET Core startup.cs file with a list of override-able events with begin/end etc. If there was any way to pass a dotnet argument to activate these as a trigger would also go a long way although currently we could pass a config file via additional files to trigger this. The reason for needing a trigger is so that these events don't occur when the developer is using the analyzer in their IDE but can be triggered when running in CI/CD pipelines.

We don't want to use the Sarif output as we want to model the data into our own format and also extract data we don't natively get from the Sarif format such as the SuppressMessage Justification field.

[mavasani]

This would potentially be addressed by DiagnosticSuppressor API for which I already have a PR: #31603. It is an API for post analysis callback, where you can have access to all reported analyzer and compiler diagnostics of registered diagnostic IDs. It is intended for suppression, but would also serve this scenario.

[jmarolf]

... extract data we don't natively get from the Sarif format such as the SuppressMessage Justification field.

@Cyassin is that the main reason you can't use /errorlog? If we added that information to the sarif log would that unblock you?

[Cyassin]

@jmarolf That would unblock me by providing access to the data I need currently. I think it would still be good to consider the feature request as outputting to a user specific format could be done rather than pick up the sarif log and massage it. It would also allow logic to help filter what is exported in the sarif instead of reporting all diagnostics from the compilation.

[mavasani]

@Cyassin - DiagnosticSuppressor API was added for Dev16.3-preview1 with #36067. You should be able to use the above suggestion to accomplish your goal. Closing the issue, kindly re-activate if that doesn't work for you.