[Package Validation] We should support excluding tfms when running the validation

[safern]

In dotnet/runtime we have been following a convention with our packages where we drop TFMs that are out of support from our packages. This results on a lot of errors that need to be suppressed because we are intentionally dropping these TFMs.

Also, when this happens, in baseline validation, if an asset is drop it will take another asset that is compatible for that TFM in the new package, i.e, if we had netcoreapp3.1 and netstandard2.0 asset, then we drop netcoreapp3.1, and now it will compare the netcoreapp3.1 asset from the baseline package vs the netstandard2.0 from the new version, resulting in a lot of errors as the API surface might be fairly different, so that comparison is not valuable in a lot of cases and it can become very noisy.

I suggest we add some support like an ItemGroup to specify ignored tfms for baseline validation, i.e:

<PackageValidationBaselineIgnoreTfm Include="netcoreapp3.1" />

Or we could introduce a flag to ignore out of support tfms and then we can use the list from the SDK for which tfms are out of support, i.e:

<PackageValidationBaselineIgnoreEolTfms>true</PackageValidationBaselineIgnoreEolTfms>

cc: @ericstj @joperezr

[joperezr]

While I agree this will become more common as time goes on and TFMs continue to go out of support, I have a few thoughts in terms of the design of the feature. Here are some of the things in my mind:

• Technically we already have support for this by just baselining the error for that specific suppression, which you can easily just add a comment on top of the specific suppression and comment that it is due to the TFM going out of support. That said, I also realize that due to the fact that the suppression file is auto-generated, you might lose the comment if the file is re-generated later.
• The problem with having this suppression via Item or property, means that the consumer project might have suppressions for a lot of TFMs that will no longer warn after the baseline version is updated and no longer has the error.
• Regarding your comment that: resulting in a lot of errors as the API surface might be fairly different, so that comparison is not valuable I actually think it is great that when doing this, you can basically get the preview of the impact that your user's targeting that dropped TFM will get after they upgrade. For instance, we just dropped some TFMs from our dotnet/iot packages, and in our case PackageValidation helped a lot measuring the kind of impact that consumers would see once they upgraded their package.

In general, I'm fine with adding this feature. As for your options in terms of execution, I probably wouldn't have a flag for <PackageValidationBaselineIgnoreEolTfms>true</PackageValidationBaselineIgnoreEolTfms> as that would add some unnecessary complexity like:

• How/When do we calculate if a TFM reached EOL?
• How can we make builds reproducible across time if we do a live check? For instance, if you build a commit hash today, vs building the same hash a year from now would potentially log different warnings, which I'm not sure we want.

[safern]

Thanks for the input. I completely agree that it is helpful to get errors and evaluate the impact and that is why I would lean torwards and itemgroup which at the end is like using NoWarn.

Technically we already have support for this by just baselining the error for that specific suppression, which you can easily just add a comment on top of the specific suppression and comment that it is due to the TFM going out of support. That said, I also realize that due to the fact that the suppression file is auto-generated, you might lose the comment if the file is re-generated later.
Regarding your comment that: resulting in a lot of errors as the API surface might be fairly different, so that comparison is not valuable I actually think it is great that when doing this, you can basically get the preview of the impact that your user's targeting that dropped TFM will get after they upgrade. For instance, we just dropped some TFMs from our dotnet/iot packages, and in our case PackageValidation helped a lot measuring the kind of impact that consumers would see once they upgraded their package.

The problem is not the error that the tfm is suppressed, the problem is when we fallback to the latest compatible asset for an asset that was dropped, like in the example that I provided. For example, take we have a netcoreapp3.1 asset that has an API A.B that is only part of netcoreapp3.1+, but we also have a netstandard2.0 asset which doesn't contain that API. Then in the new version of the package we drop the netcoreapp3.1 asset, and now when running baseline validation we will compare the netcoreapp3.1 tfm from the old package vs the netstandard2.0 asset on the new package, since it is the best compatible asset for netcoreapp3.1, that would result on a PKV0006 warning, and it will also result on a CP0002 warning for A.B API. I think that A.B warning can get pretty noisy in the suppression file, so instead of suppressing all APIs that were lost because of the dropped asset, I would just declare:

<PackageValidationBaselineIgnoreTfm Include="netcoreapp3.1" />

The problem with having this suppression via Item or property, means that the consumer project might have suppressions for a lot of TFMs that will no longer warn after the baseline version is updated and no longer has the error.

I don't imagine this feature being used as default values, but rather intentional opt-in from the user using the property or the item, so the workflow would be, I drop the tfm, I see the errors I get, I think the impact makes sense, but rather than writing a lot of suppressions into the suppressions file, I just include the tfm as part of the item group.

How/When do we calculate if a TFM reached EOL?

I believe the SDK has a list of EOL TFMs for the current .NET SDK Version, so i.e, the .NET 6 SDK knows about 1.0, 1.1, 2.0 and 2.1, then .NET 7 SDK will have those, plus 3.1 and 5.0. The .NET SDK already warns if you target and EOL tfm, so that is why I was thinking we could benefit from that list, but now that I give it a thought about your comments, I think it makes more sense for the user to explicitly list the TFMs to ignore as that is more intentional.

[joperezr]

The problem is not the error that the tfm is suppressed, the problem is when we fallback to the latest compatible asset for an asset that was dropped, like in the example that I provided.

Yeah but that is what I mean that really shows you what your customers will see if they target the dropped framework. For example, in your particular example you may have dropped netcoreapp3.1 thinking there wouldn't be many fallbacks, but by getting all of these warnings (like the CP0002 on A.B) will let you know exactly what will folks that target netcoreapp3.1 and upgrade to your new package will see. I think all of those are specially relevant to be able to see when a dropped asset result in a different asset being picked, cause you may think you still support a tfm but not realize all of the issues that come with that. Anyway, I think it is fine to have a feature that let's you opt-out of getting all of those diagnostics if you don't care, I guess I just don't think they are noise.

[dotMorten]

It would also be nice if we could override the baseline tfm. Ie if I drop .net5 and add .net6, I could tell the baseline validator to compare my current .net6 with the old .net5 APIs and ensure that when people upgrade, the only breaking change is the TFM itself.
This is especially the case for Xamarin.Android and Xamarin.iOS where we are now moving towards net6.0-android and net6.0-ios, but besides that want to make sure the API didn't actually change.

[joperezr]

Mmm that feels like a bit of a specific use case to me, so I'm not sure if this is something that should be inside package validation. There are two different layers of the Package Validation feature today:

• There is the PackageValidation layer which is the thing that understands NuGet packages, knows how to parse them, and given two packages it knows which assemblies should be compared against each other for compatibility, which then calls into the second layer which is...
• ApiCompat itself. This isn't aware of any NuGet concepts, it only takes two inputs, a left and a right, and knows how to check if the two are compatible.

Today, this feature is only exposed via PackageValidation, but the plan (and I believe this is still in scope for .NET 7 cc @ViktorHofer) is to have also an MSBuild tasks and targets to expose the second layer directly so it could be called from users targets. I think your scenario is a special case, so instead of adding this feature as a characteristic of the first layer (package validation) I think we should enable you to be able to easily run ApiCompat via targets directly for which you can then select whichever two assets you want from two different packages in order to perform the comparisons. @dotMorten thoughts?

[dotMorten]

That could probably work, but I'm not sure I agree it is specific. Dropping older targets and replacing with new ones in major versions is pretty common.

[joperezr]

I guess that is fair. I probably chose my words incorrectly, I guess my thoughts were more that doing so is effectively breaking your previous package, whether that is common or not, and to me Package Validation's layer purpose was more to be objective and find all breaking changes between two packages. That said, your point is valid that it is becoming more common to drop support for some frameworks, so I think it is fine to consider adding this extensibility point.