Merging framework references with roll forward LatestMinor and LatestMajor

[vitek-karas]

This issue is about the new roll forward feature. High level design document is runtime binding, detailed technical design is in framework version resolution. The current implementation is in progress in #5691.

The feature introduces new roll forward settings LatestMinor and LatestMajor which are intended for component loading scenarios (native hosting, COM, WinRT). Given a native application which dynamically loads some .NET Core components (for example COM objects), the first .NET Core component to load will determine the versions of shared frameworks (and thus runtime) to be used by the process (only one version of CoreCLR is allowed per-process). Any subsequently loaded .NET Core component will have to use the versions selected by the first one, if they're not compatible the component will fail to load. To increase the chances of components working well together, the first component should select the highest compatible framework versions so that the other components have higher chance of being able to run on the same framework.

For this purpose the LatestMinor and LatestMajor roll forward policies are introduced. They will select the highest available framework version. In case of LatestMinor it will be the highest version in with the same major version, for LatestMajor it will be the absolutely highest version available.

One of the impacted scenarios is merging framework references. For example:

Component
-> Microsoft.WindowsDesktop.App  3.0.0  rollForward=LatestMinor

Microsoft.WindowsDesktop.App (3.1.0)
-> Microsoft.NETCore.App         3.1.0  (Default rollForward=Minor)

On the machine only Microsoft.WindowsDesktop.App 3.1.0 is available, but the machine also has Microsoft.NETCore.App 3.1.0 and Microsoft.NETCore.App 3.2.0.

The question is what version should be selected for the Microsoft.NETCore.App.

Option 1 - propagate setting across references

In this case the rollForward=LatestMinor setting would propagate from the Component to the Microsoft.WindowsDesktop.App and since that framework doesn't explicitly specify any setting, it would apply to the reference to Microsoft.NETCore.App as well. The outcome would be that the highest available version of Microsoft.NETCore.App 3.2.0 is selected.

• Upside is that the behavior is consistent regardless if Component has direct reference to Microsoft.NETCore.App or not.
• Downside is that the component effectively overrides the default behavior for Microsoft.WindowsDesktop.App. Previously the same version of Microsoft.NETCore.App would be selected almost always, with the new behavior a newer version may be selected.
• Another downside is what should happen for "old" roll forward policies (LatestPatch, Minor, Major). Should those also propagate? If so it would probably mean changing existing behavior for many scenarios.

Option 2 - prefer closest version policy

In this case the setting would not propagate over framework reference (existing behavior). So if the Component doesn't declare direct reference to Microsoft.NETCore.App the existing behavior would be used and version 3.1.0 would be selected.
The Component would have to declare a direct reference like this:

Component
-> Microsoft.WindowsDesktop.App  3.0.0  rollForward=LatestMinor
-> Microsoft.NETCore.App         3.0.0  rollForward=LatestMinor

Note that having a direct reference to a framework which is already referenced indirectly is currently optional and thus typically not present.

The algorithm would have to merge the reference from the app 3.0.0 LatestMinor with the reference from Microsoft.WindowsDesktop.App which is effectively 3.1.0 Minor.
Selecting the policy which prefers closest version which would mean to merge the reference to 3.1.0 Minor as Minor will chose the lowest compatible version of the framework.
So in the sample above the version Microsoft.NETCore.App 3.1.0 would be selected.

• Upside is that no propagation would occur and there would be no problems with existing policies and their lack of propagation.
• Another upside is that frameworks would get predictable behavior for their dependent frameworks.
• Another upside is that not having the direct reference form the component would (in most cases) not change the outcome.
• Downside is that the component would not load the highest compatible version of Microsoft.NETCore.App and thus potentially block other components from getting loaded into the process.

Option 3 - prefer latest version policy

Just like in option 2, there would be no propagation and the change would only occur if the Component had a direct reference to the Microsoft.NETCore.App.

In this option the algorithm would use policy which prefers the latest version and thus would merge the reference 3.0.0 LatestMinor with 3.1.0 Minor into 3.1.0 LatestMinor.
In the end it would select version Microsoft.NETCore.App 3.2.0.

• Upside is just like in option 2 that there's no propagation over framework references
• Another upside is that the highest compatible version is selected and thus potentially more components will be able to load into the same process.
• Downside is that frameworks don't get predictable behavior as they might run against higher versions of their dependent frameworks
• Downside is that the behavior depends on the presence of direct reference from the component (which is otherwise optional)

Other considerations

For .NET Core 3.0 the above has no real impact as for now we're planning to always ship all frameworks in lock-step, that is it should not happen that the machine has different set of versions for Microsoft.NETCore.App and for Microsoft.WindowsDesktop.App.
We don't allow 3rd party frameworks, and thus if we always ship frameworks in lock-step, the problem will never manifest itself. This means that we may be able to change the behavior in the future without risk.

Due to limitations in native hosting, it's also very likely that most components will only depend on the root Microsoft.NETCore.App. Currently hosting is only able to load frameworks when it starts CoreCLR, once the CoreCLR is running in the process, additional frameworks cannot be added to the process. So either all components would have to depend on the higher level framework, or the process might run into ordering issues (if a component with reference only to the root framework gets loaded first, subsequent loads of components with higher level frameworks will fail). Components which only depend on the root framework don't have the above described problem.

[vitek-karas]

/cc @sdmaclea @elinor-fung @AaronRobinsonMSFT
@richlander @nguerrera @jeffschwMSFT

[AaronRobinsonMSFT]

So if the Component doesn't declare direct reference to Microsoft.NETCore.App the existing behavior would be used and version 3.1.0 would be selected.

I prefer option 1 with the above "bug" fixed. It seems we have a very ad-hoc approach to policy in this area. In many instances we require explicit references and paths which makes things difficult but knowable. In this particular domain there is so much implied policy it is difficult if not impossible to ascertain or build a stable mental model about "what will happen?". My suggestion here would be all Components must explicitly declare references and policy defined on the Component would flow to those references.

Without the above, I don't see how we have made the versioning story for users better by having so much implied policy. Be explicit and dogmatic instead of complaining about a system that is "magic" would be my suggestion here. Also, I don't see any reason that if there are scenarios where this approach doesn't work we can't provide an override, but my point is the most common user scenario should be the easiest to understand and debug.

[vitek-karas]

@AaronRobinsonMSFT I do agree that it's better to be explicit about intent/requirements. Unfortunately it's not the precedence in this area. In general our way of specifying framework dependencies has been mostly "make it work by default". And combined with the other approach "SDK will take care of doing the right thing, so it kind of doesn't matter how it's expressed in runtime config".
Components are somewhat special because they don't have a good SDK story (hopefully yet) and by nature they will need to be a bit more explicit about their intent (of being components).

That said we need the system to work reasonably well even if the component doesn't declare all references.

I'm curious: On one hand you advocate for option 1 which is basically "it doesn't matter if the component doesn't declare all references" and at the same time you advocate for all components to declare all references. We can definitely do both, I'm just wondering what is the priority on this (from your point of view).

As noted above option 1 has a potential issue in it: The existing settings in 2.* don't implement the propagation described in option 1. So if we did the simple thing and implemented the propagation for everything, it will likely introduce quite a few changes to behavior of 2.* apps.

Possible solution could be (and this has been proposed during PR discussions by @sdmaclea ):
Internally separate the Latest and Minor parts of the LatestMinor.

• Minor/Major/Patch combined with the version in the framework reference effectively define a version range. So one part of a framework reference is version range.
• The other part, orthogonal, is "closest" or "latest". That defines if when searching for a framework, how to pick one version of potentially many compatible matches.
  • "closest" means find the lowest compatible version (the one closest to the minimu version specified)
  • "latest" means find the highest compatible version

All existing policies (in 2.*) are always "closest" and some version range. So introducing "latest" is not a breaking change since that will only ever happen by using the new settings.

We could make it so that the "closest"/"latest" does propagate across framework reference, but version ranges don't (it actually makes lot of sense thinking about it now). Then it's just about "merging" the "closest"/"latest" flags incoming from higher level with those in the config being processed. For this it feels that "latest" should always win over "closest".

The downside to this approach is that it's not easy to explain:

• LatestPatch - does not actually propagate latest in the same way (we can't for backward compat reasons). It really means literally what it says - and we apply it on top of all the other settings (we always pick the latest patch, unless you disable the feature through Disable).
• The "closest" is sort of hidden - on the other hand it's been how the system worked so far.

That said I think I prefer this approach

• It seems like the one where components will most likely actually work
• It doesn't require references to all the frameworks

[AaronRobinsonMSFT]

On one hand you advocate for option 1 which is basically "it doesn't matter if the component doesn't declare all references" and at the same time you advocate for all components to declare all references. We can definitely do both, I'm just wondering what is the priority on this (from your point of view).

@vitek-karas I was going for option 1 based on the title of "propagate setting across references". In my mind how versions are respected should be on the top level component and then flow from there. For example ASP.NET and WindowsDesktop are both high level components that seem to reference NetCoreApp. This means that the highest level component on the stack defines versioning and that behavior is inherited by all components below that. My understanding for terms like "framework" and "component" could be off or I am missing something major.

For what its worth, without a large support matrix and several examples I don't think this is any easier to understand from a consumer. I would attempt to restrict some of the loose behavior if possible. I realize this is worth in the SDK and could cause heck with roll-forwards from 2.0, but I think in the long run it would be worth it.

[vitek-karas]

Currently agreed upon solution is:

• Separate the "latest" bit in LatestMinor and LatestMajor - so that internally it's represented as version range (patch, minor, major) and latest bit.
• The latest bit is merged such that latest always wins
• The latest bit is propagated into frameworks, so that if the Component specified latest, then all referenced from frameworks are now also treated as latest.
• version range is merged separately from the latest bit.

To solve the problem of unexpected upgrades to framework dependencies, first party frameworks will explicitly specify roll forward behavior. Both Microsoft.AspNet.App and Microsoft.WindowsDesktop.App will refer to Microsoft.NETCore.App with LatestPatch roll forward behavior.

[vitek-karas]

Changes implemented in dotnet/core-setup#6569 .