Enable integration with 3rd party assets management solutions

[mkArtakMSFT]

Consider productionizing Microsoft.AspNetCore.ClientAssets package

[javiercn]

Here is a draft of the design:

Productionizing the client assets package

There are several challenges integrating third-party tools that produce or transform web assets as part of the build process:

• Developers need to write MSBuild targets to invoke their tools.
• Developers do not generally run their custom targets at the right point during the build.
• Developers generate the output into the wwwroot folder of the app, which then causes it to be considered as an input in the next build.
• Developers don't account for inputs and outputs for the tools, which means the tools always run and add time to the build even if the outputs are up to date.

Goals

• Simplify the integration of third-party tooling with static web asssets.

Out of scope

• Solutions for complex scenarios like dealing with NPM workspaces and other mono-repo setups.
• Orchestrating invocation in sequence for third-party dependency tools beyond installing tools and running them.
• Watch support (if the tool has support for watch that is something we might consider in the future)

Technical design

We want to offer a declarative model where customers can indicate what tool to run, the inputs and the outputs and we take care of the rest.

We will likely need a couple of "phases" or extensibility points in the pipeline.

One initial phase to ensure that any third-party dependency toolchain (like npm or yarn) had a chance to run.

A second phase where we invoke whatever tool or script we are given.

Ideally, we want this to be a collaborative process where multiple tools can be added provided they don't depend on each other.

That allows packages to include a command for their toolchain as long as it doesn't depend on the outputs of any other tools.

One way we can make this work is by defining the invocations inside an item group as follows:

<ItemGroup>
  <ClientAssetTool Include="InstallNpmDependencies">
    <Stage>Dependencies</Stage>
    <Command>npm install</Command>
    <Directory>assets</Directory>
    <Inputs>assets\package.json;assets\package-lock.json</Inputs>
    <ExcludeInputs></ExcludeInputs>
    <Outputs>assets\node_modules\package-lock.json</Outputs>
    <ExcludeOutputs></ExcludeOutputs>
  </ClientAssetTool>

  <ClientAssetTool Include="RunNpmBuildScript">
    <Stage>Generation</Stage>
    <Command>npm run build:$(Configuration)</Command>
    <Directory>assets</Directory>
    <Inputs>assets\**</Inputs>
    <ExcludeInputs></ExcludeInputs>
    <Outputs>*.js</Outputs>
    <ExcludeOutputs>*.map.js</ExcludeOutputs>
  </ClientAssetTool>
</ItemGroup>

The metadata included in the item group defines all the aspects of how and when the tool needs to run:

• Stage indicates when to run the tool:
  • Dependencies: This stage is intented to restore/install/retrieve required packages.
    • I want to avoid calling it Restore to avoid confusion as this will only run as part of build.
  • Generation: This stage is intended to run whatever commands need to run to generate the asset outputs.
    • If a set of tools need to be run sequentially the correct approach is to use a script, we won't handle ordering the different tools meant to run at a given stage.
• Command indicates the command to run.
• Directory indicates the directory from where to run the command from.
• Inputs, ExcludeInputs: Allows defining patterns to compute the inputs for a tool.
• Outputs, ExcludeOutputs: Allows defining patterns to compute the outputs for a tool.

With this metadata we will do as follows:

• Add a Target to static web assets:
  • ResolveGeneratedStaticWebAssetsInputs that will run before resolving assets for the current project.
• Create a new task RunClientAssetsTool that handles executing the client assets commands:
  • Resolves the inputs and outputs for a given entry.
  • Determines if it needs to run the tool.
  • Runs the tool.
  • Captures the outputs.
  • Returns the outputs.
• The Dependencies stage doesn't add any output to the build (only captures them for incrementalism).
• The Generation stage adds the outputs as ContentItems with the Link to the wwwroot folder.
• During each stage, we log in the inputs, outputs, and the decissions we make along the way to make troubleshooting inside MSBuild easy.

Notes on testing

• We don't have to use any third-party tool for testing this, we can use a cmd/powershell/bash script as long as we make it do whatever is needed to ensure it covers the scenarios we care about.

The biggest part in this aspect is the incrementalism. Some scenarios that I can enumerate:

• Things that should trigger the tool to run:
  • An additional input was added.
  • An existing input was removed.
  • An existing input is newer than the last outputs.
  • Any information about the tool changed (command, input pattern, output pattern, exclusions, directory, etc.)
  • No outputs were found.
• Things that should not trigger the tool to run again:
  • Inputs are the same and older than outputs.
  • Outputs didn't change after the tool ran (even if they are older).

Questions

• Do we need a separate stage for preparing production level assets?
  • SPAs currently generate the production bundle only during publish.
  • Static Web Assets already supports the concept of Build and Publish assets, so we might want to levarage it here.
• How do we pass the output folder information to a third-party tool?
  • We want to put the outputs in the intermediate output folder obj\clientassets\toolname to avoid them confliciting with the build.
  • We also want to support tools that put the output within the "source" tree and prevent those outputs from becoming inputs in the next build.
  • We can pass the output path as an environment variable, but some tools might only take command-line arguments.
    • We could have a token that we replace in the command before running it, like [TOOL_OUTPUTDIR] in addition to setting an environment variable when running the command.
• Do we want to define the generated assets as static web assets directly or do we want to mark them as content.
  • I believe it is better to mark them as Content and use the Link functionality to define their relative paths in the project. Static web assets already takes that into account when discovering new assets and that automatically gets them to participate in other parts of the pipeline, like scoped CSS or JS modules. Alternatively, features like scoped CSS and JS modules would need to look at an additional item group.
• Do we want to provide conventions for popular tools like NPM?
  • These can be on the SDK or in packages.