csc caching for file-based apps doesn't consider non-.cs default items, e.g. *.razor, leading to false cache hits

[DamianEdwards]

The optimization to dotnet run for file-based apps that avoids calling through MSBuild does not play well with apps that include non-.cs default items, e.g. file-based apps using the web SDK that have .cshtml or .razor files. These files will be auto-included as items by the SDK targets due to file-based app's support for default items, but changes to the set of files during the inner-loop will not result in a cache-miss (e.g. changing a file name, adding a new file, etc.) and thus runs will be out-of-date with the latest source until a run is done with the --no-cache flag.

We should consider options to improve the situation. We could try and disable the caching when non-default but "compile adjacent" items are detected (not sure how, but the point is files like *.cshtml and *.razor emit .cs files via a source generator, hence they affect compilation but aren't <Compile /> items), or, even better, figure out how to robustly include them in the cache calculation (again, no idea how, this one might be impractical as they're added via globs in the SDK which would need to flow through to dotnet run so that it could re-evaluate whether the result of the glob has changed between runs, and at that point it feels like we're likely over the line of doing too much that MSBuild already does, just to avoid it). Perhaps it's ok to special-case *.cshtml and *.razor when the Web/Razor SDK is detected given how tied they are to the compiler now anyway.

/Cc @baronfel @jjonescz

[jjonescz]

but changes to the set of files during the inner-loop will not result in a cache-miss (e.g. changing a file name, adding a new file, etc.)

I think the current optimizations don't even consider changes to contents of those files.

We should consider options to improve the situation.

One thing I wanted to look at is dotnet watch - it also needs to detect changes to files, and presumably it queries MSBuild's Project instances to get a list of "compilation inputs" to determine files to watch and we could do the same.

One problem here is the "simple CSC-only" optimization. Currently it checks there are no #: directives and implicit build files and if not, it invokes CSC with hard-coded arguments. If we also needed to check there are no included files via some glob, that might slow things down... Although we already have the optimization to not include anything if there is a .*proj file nearby, which we could check here as well, so hopefully the glob check would be fast in common scenarios.

But note that we would need to do this glob check every time - otherwise we wouldn't detect new files being added.

[jjonescz]

Btw, there is currently an opt out (#:property FileBasedProgramCanSkipMSBuild=false) that should opt you out of all run-file caching, so if you want to have a correct .razor experience, you could specify that (or SDKs, including the .NET Web SDK, could specify that).

Actually, that wouldn't opt out of the --no-build optimization, but we should probably fix that (so the property works as opt out for that too).

[DamianEdwards]

#:property FileBasedProgramCanSkipMSBuild=false

Oh perfect, was going to ask about that!

Actually, that wouldn't opt out of the --no-build optimization, but we should probably fix that (so the property works as opt out for that too).

Yep, agreed.

I think the current optimizations don't even consider changes to contents of those files.

Ah so if you only change content of those files, you won't even get a call to csc. But if you change content of those files and a .cs file, then csc would be re-invoked with same args which should result in them being regenerated by the source generator and thus updated.

[jjonescz]

Actually, that wouldn't opt out of the --no-build optimization, but we should probably fix that (so the property works as opt out for that too).

Yep, agreed.

Here is the fix for that:

• Make file-based caching opt out work for the basic up-to-date check too #50927