Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy local intellisense xmls for assemblies with source of truth. #79134

Merged
merged 27 commits into from
Feb 1, 2023

Conversation

carlossanlop
Copy link
Member

@carlossanlop carlossanlop commented Dec 2, 2022

This PR explores obtaining the intellisense xml files for assemblies that have their source of truth in triple slash from the local build, and the rest of the assemblies get their intellisense xml files from the nuget package that comes from the docs repo.

Made sure the dotnet pack command also included the correct intellisense file in the nupkgs.

@ghost
Copy link

ghost commented Dec 2, 2022

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Tentative draft. Co-authored with @smasher164

This PR explores obtaining the intellisense xml files for assemblies that have their source of truth in triple slash from the local build, and the rest of the assemblies get their intellisense xml files from the nuget package that comes from the docs repo.

Author: carlossanlop
Assignees: carlossanlop, smasher164
Labels:

area-Infrastructure-libraries

Milestone: -

@smasher164
Copy link
Member

The CI errors seem to be happening because it's trying to copy the XMLs over on configurations besides -allconfigurations, which shouldn't happen. Maybe the docsFile target runs even when not in allconfigurations?

@carlossanlop
Copy link
Member Author

Our assumption was that we only want this xml copy to happen when -allconfigurations is executed because when we do not build -allconfigurations, the only folder that gets created for many assemblies is the net7.0-windows one, but we need the non-OS specific one for this to work. We could be wrong about our approach, though.

If we confirm we only want this copy to happen in -allconfigurations, then we need to find a way to detect that the user passed -allconfigurations and use it as a condition to run our target.

@ghost ghost closed this Jan 2, 2023
@ghost
Copy link

ghost commented Jan 2, 2023

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.

Directory.Build.props Outdated Show resolved Hide resolved
Directory.Build.targets Outdated Show resolved Hide resolved
eng/packaging.targets Show resolved Hide resolved
eng/packaging.targets Show resolved Hide resolved
eng/restore/docs.targets Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
eng/intellisense.targets Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
@carlossanlop carlossanlop marked this pull request as ready for review January 10, 2023 17:20
eng/intellisense.targets Outdated Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
@ericstj
Copy link
Member

ericstj commented Jan 12, 2023

Things to test to make sure this is behaving correctly:

  1. Library nuget package is honoring this flag and packing source built XML vs doc-team provided XML. Check both cases.
  2. Microsoft.NETCore.App targeting pack is honoring this flag. Check both cases.
  3. Transport packages are honoring this flag. Check both cases.

I don't think any of these are broken, but I'm not sure about the last one. It would be good to check.

@ViktorHofer
Copy link
Member

I just submitted dotnet/arcade#12188 to make it possible to have a compiler error for NoWarn=CS1591. For historical reasons that error is globally suppressed in all Arcade.Sdk consuming repositories.

We should set <SkipArcadeNoWarnCS1591>true</SkipArcadeNoWarnCS1591> in Versions.props and then set <NoWarn Condition="'$(UseIntellisenseDocumentationFile)' != 'true'">$(NoWarn);1591</NoWarn> in intellisense.targets to not break existing projects that haven't yet annotated all their public API.

Directory.Build.props Outdated Show resolved Hide resolved
</Choose>

<!-- TODO: Remove this target when no library relies on the intellisense documentation file anymore.-->
<Target Name="ChangeDocumentationFileForPackaging"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we just discussed, this might be too late to impact the result of a ProjectReference.

ProjectReferences (as used in the transport packages) only recieve the path to the dll in the bin folder and find the doc file relative to that.

To support ProjectReference we may want to swap the value of @(DocFileItem) instead and do so right before CopyFilesToOutputDirectory. This will make it so that the "shipping" xml is always next to the binary in the bin folder. The compiler generated one (in the case it is not shipping) will remain in the obj folder.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a sample that does this: ericstj@94849ba

Copy link
Member

@ViktorHofer ViktorHofer Jan 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you verify that this works in a pack-only scenario as well? dotnet pack --no-build

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some tests in a comment in the main thread.

@carlossanlop
Copy link
Member Author

I just submitted dotnet/arcade#12188 to make it possible to have a compiler error for NoWarn=CS1591

Thanks, @ViktorHofer. I'll rebase the PR.

@carlossanlop

This comment was marked as outdated.

Build.proj Outdated Show resolved Hide resolved
@carlossanlop
Copy link
Member Author

Pending trying to find out why Brotli is only generating SR files (Viktor is helping me).

@ericstj
Copy link
Member

ericstj commented Jan 27, 2023

The compiler isn't going to emit those for you, you'll need to merge them from the dependent assembly. Maybe through a build task or source generator or something? @safern might remember some thinking around that.

@ViktorHofer
Copy link
Member

ViktorHofer commented Jan 28, 2023

Pending trying to find out why Brotli is only generating SR files (Viktor is helping me).

Broti only has SR resources because the net8.0 tfm is a PNSE assembly:

<GeneratePlatformNotSupportedAssemblyMessage Condition="'$(TargetPlatformIdentifier)' == ''">SR.IOCompressionBrotli_PlatformNotSupported</GeneratePlatformNotSupportedAssemblyMessage>

That means that it will automatically generate public API with throwing body based on the contract/reference assembly's source. The reference assembly's source isn't XML doc annotated (by design). We will probably need a design discussion first how to handle this. In #44969, Santi talks about that problem a bit.

IMHO ideally (because that would also solve some other scenarios like the ref <--> src project convergence) we would stop generating PNSE source code automatically and use checked-in code instead. I would imagine that we could either

  • migrate existing PNSE auto-generated code into runtime condition (i.e. if (OperatingSystem.IsWindows) ..., else if (...), else { throw PNSE(); }
  • or if we want to keep building for different platforms, use compiler preprocessor directives:
/// <summary>
/// This is a public api method
/// </summary>
public void PublicApiMethod()
{
#if WINDOWS
    WindowsImpl();
#elif UNIX
    UnixImpl();
#else
    throw new PlatfornNotSupportedException(SR.X);
#endif
}

These are just some random thoughts from my side. I strongly support any design that moves away from auto-generated PNSE code as that will also solve other hard problems: #58163.

@ViktorHofer
Copy link
Member

ViktorHofer commented Jan 28, 2023

from @ericstj:

The compiler isn't going to emit those for you, you'll need to merge them from the dependent assembly. Maybe through a build task or source generator or something? @safern might remember some thinking around that.

It would be great if we could find a solution that would contribute to #58163 as well:

from #58163:

It's yet unclear how to generate these type forwards without having a reference assembly available that represents the public API surface area. A possible solution to that could be to check in the type forwards, adding them to the compilation input and making sure that they are kept up to date. The problem with that is that Roslyn keeps passed-in type forward attributes in reference assemblies instead of following the type and replacing the attribute with the actual type. We should sync up with the Roslyn team to see if supporting this scenario is feasible.

Overall, I see #58163 and this effort tangled. By choosing the right design, we should be able to contribute to both efforts at the same time.

That said, to not block this PR, we should be able to check-in the improvements made here by continuing to use the intellisense docs files in projects that are blocked. Blocked means that the projects uses any of these features:

  • The NetCoreAppCurrent configuration uses the GeneratePlatformNotSupportedMessage feature or
  • The NetCoreAppCurrent configuration uses the IsPartialFacadeAssembly feature.

@safern
Copy link
Member

safern commented Jan 30, 2023

Joining the party a little late but on Friday I did not get a chance to reply.

When we were looking at this a couple of years ago and based on the discussion above, I think there are 3 main scenarios to consider for generating the XML files:

  1. PNSE assemblies which use the GeneratePlatformNotSupportedMessage feature.
  2. Facade assemblies which have at least one typeforward.
  3. Cross RID assemblies which have different runtime implementations.

I think we only had an idea on how to solve for 2 and 3.

For facade assemblies and Cross RID assemblies we said that we were going to add an MSBuild Task that would run ONLY on the outer build to not affect the vertical build times and also we need the outputs of all the OSs. The MSBuild Task would do something like:

  1. For facade assemblies:

    • get all the typeforwards generated (maybe choose the built asset for the matching TargetOS or the Host OS, since we are in an outerbuild), get the generated XML file with the docs for all it's references and get the matching docs for the types that were forwarded and move them over to the facade assembly's main doc xml.
    • I think this could also be done in a source generator, but that would run for all inner builds, but might be "simpler" implementation wise?
    • I don't know if this could be a Roslyn feature that could be built via an opt-in switch, something like, GenerateXmlDocsForTypeForward and also how feasible since Roslyn would need to resolve the docs for the forwarded types.
  2. For cross RID assemblies we agreed that we were going to turn on the doc generation and also the warning that all public API must have a triple slash document so that we ensure all the OSs implementations had triple slash comments, however we need to include some validation to make sure we produce the same XML output regardless of the OS it is being built on, or this could be optimized to only be done when in a local developer build or an all configuration build to not affect the CI verticals. The task would do something like:

    • Validate all xml files for all OS builds are the same. I don't know if this could just do a git diff (that would require all contributors to have git installed on the path, but would perhaps be simpler and more efficiente than doing it on an MSBuild Task? It could also be text diff if that meets the criteria.
    • Choose any produced xml file (perhaps the one applicable to the Target OS/Host OS) as we already ensured that all of them are the same.

For the PNSE scenario I see 2 options:

  1. Check-in the XML file with all the documentation for the APIs and perhaps add validation that all the public APIs in the assembly are present in the XML file.
  2. As Viktor mentioned above, remove the magic PNSE generator (that would potentially be needed if we ever wanted to get RID of the ref assemblies' source code) and check-in the source code that throws the PNSE explicitly, if it does for all OSs then a simple PNSE source file, if not, it could be included via compile conditions on OS/TFM and then the checked-in source code would need to have the triple slash comments.

But I think these are incremental steps that could be done separately, and you could potentially start just having the source of truth being triple slash for assemblies that don't use IsPartialFacadeAssembly and don't use GeneratePlatformNotSupportedMessage.

@ViktorHofer
Copy link
Member

ViktorHofer commented Jan 31, 2023

@carlossanlop as we discussed offline, please add a validation target in intellisense.targets that emits an error if UseIntellisensePackageDocXmlFile is false (means that the project opts into using the compiler produced xml file) and the project either sets IsPartialFacadeAssembly to true or when GeneratePlatformNotSupportedAssemblyMessage isn't empty as we currently don't support projects that use these features.

…current project, and throws errors if the UseIntellisensePackageDocXmlFile is set to false and the assembly is either a partial facade or uses PNSE.
@carlossanlop
Copy link
Member Author

Below is a test of the latest 2 commits.

The error message shows up in assemblies where I temporarily set UseIntellisensePackageDocXmlFile to false, and they are either partial facades or use PNSE. The only one that does not have neither of those conditions is Cbor. It's the first result, built successfully, and is the only one we will merge with UseIntellisensePackageDocXmlFile set to false with this PR.

PS D:\runtime\src\libraries> dotnet build .\System.Formats.Cbor\src\System.Formats.Cbor.csproj -f net8.0 /bl 
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
    Determining projects to restore...
    All projects are up-to-date for restore.
    Microsoft.Interop.SourceGeneration -> D:\runtime\artifacts\bin\Microsoft.Interop.SourceGeneration\Debug\netstandard2.0\Microsoft.Interop.SourceGeneration.dll
    LibraryImportGenerator -> D:\runtime\artifacts\bin\LibraryImportGenerator\Debug\netstandard2.0\Microsoft.Interop.LibraryImportGenerator.dll
    System.Formats.Cbor -> D:\runtime\artifacts\bin\System.Formats.Cbor\ref\Debug\net8.0\System.Formats.Cbor.dll
    System.Formats.Cbor -> D:\runtime\artifacts\bin\System.Formats.Cbor\Debug\net8.0\System.Formats.Cbor.dll

  Build succeeded.
      0 Warning(s)
      0 Error(s)

  Time Elapsed 00:00:01.31

PS D:\runtime\src\libraries> dotnet build .\System.Numerics.Vectors\src\System.Numerics.Vectors.csproj -f net8.0 /bl          
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
  D:\runtime\eng\intellisense.targets(14,5): error : Cannot currently generate full xml documentation for an assembly that is a partial facade. [D:\runtime\src\libraries\System.Numeric
  s.Vectors\src\System.Numerics.Vectors.csproj]

PS D:\runtime\src\libraries> dotnet build .\System.IO\src\System.IO.csproj -f net8.0 /bl          
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
  D:\runtime\eng\intellisense.targets(14,5): error : Cannot currently generate full xml documentation for an assembly that is a partial facade. [D:\runtime\src\libraries\System.IO\src\
  System.IO.csproj]

PS D:\runtime\src\libraries> dotnet build .\System.Formats.Tar\src\System.Formats.Tar.csproj -f net8.0 /bl  
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
  D:\runtime\eng\intellisense.targets(15,5): error : Cannot currently generate full xml documentation for an assembly that generates PNSE. [D:\runtime\src\libraries\System.Formats.Tar\
  src\System.Formats.Tar.csproj]

PS D:\runtime\src\libraries> dotnet build .\System.IO.Compression.Brotli\src\System.IO.Compression.Brotli.csproj -f net8.0 /bl
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
  D:\runtime\eng\intellisense.targets(15,5): error : Cannot currently generate full xml documentation for an assembly that generates PNSE. [D:\runtime\src\libraries\System.IO.Compressi
  on.Brotli\src\System.IO.Compression.Brotli.csproj]

PS D:\runtime\src\libraries> dotnet build .\System.IO.Ports\src\System.IO.Ports.csproj -f net8.0 /bl                
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
  D:\runtime\eng\intellisense.targets(15,5): error : Cannot currently generate full xml documentation for an assembly that generates PNSE. [D:\runtime\src\libraries\System.IO.Ports\src
\System.IO.Ports.csproj]

PS D:\runtime\src\libraries> dotnet build .\System.Drawing.Common\src\System.Drawing.Common.csproj -f net8.0 /bl
  MSBuild version 17.5.0-preview-23067-01+771199907 for .NET
  D:\runtime\eng\intellisense.targets(14,5): error : Cannot currently generate full xml documentation for an assembly that is a partial facade. [D:\runtime\src\libraries\System.Drawing
  .Common\src\System.Drawing.Common.csproj]

eng/intellisense.targets Outdated Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
eng/intellisense.targets Outdated Show resolved Hide resolved
…llisense... is not set to true.

Also moved the initial definition to its own solitary property group right before the verification target runs.

Rename the properties that acquire the value of a file path to "FilePath" instead of just "File", for clarity.
@carlossanlop
Copy link
Member Author

CI failure is unrelated and pre-existing: #48798
The manual tests in various csproj with and without enabling the new property, worked as expected.
Signed-off, so I'm merging now. :shipit:

@carlossanlop carlossanlop merged commit 0b3703e into dotnet:main Feb 1, 2023
@carlossanlop carlossanlop deleted the DocsXml branch February 1, 2023 22:29

<Target Name="CopyDocumentationFileToXmlDocDir"
AfterTargets="CopyFilesToOutputDirectory"
Condition="'$(IsNetCoreAppSrc)' == 'true' and '$(TargetFramework)' == '$(NetCoreAppCurrent)'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just realized that the '$(TargetFramework)' == '$(NetCoreAppCurrent)' condition results in a behavior difference and will force us to overbuild. When building the repo, by default only the nearest NetCoreAppCurrent tfm will be built. In the case of building on Windows, this means net8.0-windows or net8.0 if projects don't have a corresponding windows platform TFM.

We should remove that artificial limitation and restore the previous behavior. I could see the following cases to apply:

  • A project doesn't multi-target => use NetCoreAppCurrent condition (status quo).
  • A project multi-targets and all inner builds are built (no tfm filtering) => run this target in the NetCoreAppCurrent inner build (status quo).
  • A project multi-targets but an inner build runs stand-alone (tfm filtering) => Don't condition on a TFM.

I don't know if it's possible to determine the third case in an msbuild environment, but that's the default build behavior in dotnet/runtime and will be utilized in the VMR with RID builds. Therefore we should try to make that work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created an issue to track this: #82191

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants