[rush] Add plugin and cloud build cache with rush plugins

[chengcyber]

Summary

Fixes #2898

Rush plugin is the first step to build rush ecosystem. 😃

Details

1. Implement rush plugin
2. extract s3, azure cloud build cache to plugins

Further roadmap:

• make default s3, azure plugins to individual packages, [rush] Support for only installing Rush components needed for the current repo.  #2408
• add lifecycles to make rush more customizable, this benefits cross repo use case a lot!

Cloud build cache with plugin workflow

Rush plugin Load workflow

How it was tested

Tried to add jests but not too much, tested in local with MacOS

[elliot-nelson]

Thanks for putting this together @chengcyber! Haven't reviewed the whole thing yet but I left some initial thoughts about the API.

[octogonz]

Copy of my notes from the Rush Hour breakout meeting:

(These were informal suggestions and not final decisions.)

• Suggested to move "rushPlugins" into its own file common/config/rush/plugins.json

• Suggested that plugin options should go in individual files so we can enforce JSON schemas: common/config/rush-plugins/my-plugin-options.json

• JSON schema validation could be applied by Rush, rather than each plugin calling validateObject()

• pluginsAutoinstallerName should be specified individually for each plugin, so that users have the option to have multiple autoinstallers

• In the futur we'd like to defer loading of autoinstaller depending on whether feature is relevant. (Example: Maybe a publishing autoinstaller doesn't need to be installed unless rush publish is invoked.)

• Plugins should be described by a "manifest" JSON file, so that Rush can query essential metadata without having to eval() JavaScript code

• Philosophically, we prefer static DATA (e.g. rush.json) over executable CODE (e.g. .eslintrc.js) for several reasons: (1) DATA can be loaded extremely fast, (2) DATA doesn't use require() so it can be loaded even in a context without rush install, (3) DATA can be cached because it is deterministic, whereas a cache of require("./.eslintrc.js") would be invalid, for example if the script output depends on environment variables.

• Use Tapable 2.0 but don't upgrade Heft. See Rush's allowedAlternativeVersions setting in common-versions.json

• Use the Import API from node-core-library instead of require.resolve()

• cloudCacheProviderFactories would be better as an API function, since exposing the private Map<>() data structure provides no way for Rush to do validation or logging

[octogonz]

@chengcyber I will ready your other questions/comments and follow up separately.

[dmichon-msft]

• Suggested that plugin options should go in individual files so we can enforce JSON schemas: common/config/rush-plugins/my-plugin-options.json
  This prevents you from using multiple copies of a plugin with different configurations, so scenarios that would otherwise work that way would have to accept multiple configurations in their single options structure.

• JSON schema validation could be applied by Rush, rather than each plugin calling validateObject()
  Ideally we don't want this to happen until the plugin becomes relevant, though.

• Use the Import API from node-core-library instead of require.resolve()
  @octogonz Why? require.resolve() ensures that we use the proper node algorithm (as in will do it exactly how NodeJS would do so itself), has no external dependencies, and has supported the paths override parameter to tell it where to resolve relative to since at least Node 8.

[octogonz]

• Suggested that plugin options should go in individual files so we can enforce JSON schemas: common/config/rush-plugins/my-plugin-options.json

This prevents you from using multiple copies of a plugin with different configurations, so scenarios that would otherwise work that way would have to accept multiple configurations in their single options structure.

@dmichon-msft this is a good point. Can you think of a design that provides: (1) VS Code IntelliSense for people who are editing the config files and (2) support for rush init style file templates that have nice documentation comments explaining how everything works? I'd like for the plugin options to have the same user experience as Rush config files, versus the mysterious-bag-of-key-value-pairs that's the current experience for Heft plugin options.

Like, maybe the filenames have a suffix with an integer or instance id?

• JSON schema validation could be applied by Rush, rather than each plugin calling validateObject()

Ideally we don't want this to happen until the plugin becomes relevant, though.

Agreed. And maybe you could have a rush validate-everything command that checks all the various files for mistakes at the time when people are making changes to those files. Anyway I wasn't arguing for a specific feature, just a general principle that the JSON schema should be public and standardized, versus being a private implementation detail.

• Use the Import API from node-core-library instead of require.resolve()

@octogonz Why? require.resolve() ensures that we use the proper node algorithm (as in will do it exactly how NodeJS would do so itself), has no external dependencies, and has supported the paths override parameter to tell it where to resolve relative to since at least Node 8.

If Import.resolvePackage() or Import.resolveModule() are missing nice functionality that require.resolve() has, we should get that fixed. We designed these APIs to be better, not worse. :-)

[dmichon-msft]

@dmichon-msft this is a good point. Can you think of a design that provides: (1) VS Code IntelliSense for people who are editing the config files and (2) support for rush init style file templates that have nice documentation comments explaining how everything works? I'd like for the plugin options to have the same user experience as Rush config files, versus the mysterious-bag-of-key-value-pairs that's the current experience for Heft plugin options.

The easiest way I could think of to do this would be to have the list of plugins contain a single option for each, which is the name of the independent config file; that would allow collisions to be resolved while maintaining a fixed schema for the rush.json/rush-plugins.json file. Since we can load all the plugin configs in parallel it shouldn't be too bad on load perf.

Alternatively, is it valid to define a $schema field inside of a sub-object?

[chengcyber]

@dmichon-msft this is a good point. Can you think of a design that provides: (1) VS Code IntelliSense for people who are editing the config files and (2) support for rush init style file templates that have nice documentation comments explaining how everything works? I'd like for the plugin options to have the same user experience as Rush config files, versus the mysterious-bag-of-key-value-pairs that's the current experience for Heft plugin options.

The easiest way I could think of to do this would be to have the list of plugins contain a single option for each, which is the name of the independent config file; that would allow collisions to be resolved while maintaining a fixed schema for the rush.json/rush-plugins.json file. Since we can load all the plugin configs in parallel it shouldn't be too bad on load perf.

Alternatively, is it valid to define a $schema field inside of a sub-object?

Agreed, for now plugin configuration need specify optionsJsonFilePath explicitly to allow collisions. It might kinda inconvenience to use, since user need create options files manually and follow the plugin system convention strictly. I'd like to do some research on nested schema

Update:

Do some experiments in vscode and specifing $ref works, e.g.

{
  "$schema": "http://json-schema.org/draft-04/schema",
  "type": "object",
  "properties": {
    "plugins": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "options": {
            "$ref": "<relative_path>/some-options.schema.json"
          }
        }
      }
    }
  }
}

Had a lightbulb: we can dynamically generate a common/rush-plugins/rush-plugins.schema.json updated when rush update and commit to git, let's say we got some-plugin and other-plugin, therefore rush-plugins.schema.json will be:

{
  "$schema": "http://json-schema.org/draft-04/schema",
  "type": "object",
  "required": ["plugins"],
  "properties": {
    "plugins": {
      "type": "array",
      "items": {
        "type": "object",
        "allOf": [
          {
            "properties": {
              "pluginName": {
                "type": "string"
              },
              "options": {
                "type": "object"
              }
            }
          },
          {
            "oneOf": [
              {
                "properties": {
                  "pluginName": {
                    "type": "string",
                    "enum": ["some"]
                  },
                  "options": {
                    "$ref": "../some-options.schema.json"
                  }
                }
              },
              {
                "properties": {
                  "pluginName": {
                    "type": "string",
                    "enum": ["other"]
                  },
                  "options": {
                    "$ref": "../other-options.schema.json"
                  }
                }
              }
            ]
          }
        ]
      }
    }
  }
}

meanwhile, rush-plugins.json use this new schema and nested options schema works!

{
  "$schema": "<relative_path>/rush-plugins.schema.json",
  "plugins": [
    {
      // other properties
      "pluginName": "some",
      "options": {
        // vscode hint/autocomplete works
      }
    }
  ]
}

[chengcyber]

Update 2021/10/13

• rush plugin requires a manifest file.
• update plugin manifest when rush update
• lazy load plugins those who setup associatedCommands, and load the rest plugins eagerly
• plugin can define custom command line

Use case 1: custom cache provider

Plugin Part:

// rush-plugin-manifest.json
{
  "plugins": [
    {
      "pluginName": "rush-custom-build-cache-plugin",
      "description": "My awesome rush build cache plugin",
      "entryPoint": "lib/index.js",
      "optionsSchema": "lib/schemas/amazon-s3-config.schema.json",
      "associatedCommands": [
        "build",
        "rebuild",
        "write-build-cache",
        "update-cloud-credentials"
      ]
    }
  ]
}

You might notice associatedCommands, if specified, plugin will only be applied when associated command run.

// src/index.ts

// use import type, it will be removed in compiled file
import type { IRushPlugin, RushSession, RushConfiguration } from '@microsoft/rush-lib';
import type { MyBuildCacheProvider } from './MyBuildCacheProvider';

// for lazy load
const MyBuildCacheProviderModule: typeof import('./MyBuildCacheProvider') = Import.lazy(
  './MyBuildCacheProvider',
  require
);

interface IMyConfigurationJson {
  /// ...
}

export default class RushCustomBuildCachePlugin implements IRushPlugin {

  public apply(rushSession: RushSession, rushConfiguration: RushConfiguration) {
      rushSession.hooks.initialize.tap('rush-custom-build-cache-plugin', () => {
      rushSession.registerCloudBuildCacheProviderFactory(
        'my-custom-build-cache',
        (buildCacheConfig): MyBuildCacheProvider => {
          type IBuildCache = typeof buildCacheConfig & {
            myBuildCacheConfiguration: IMyConfigurationJson;
          };
          const { myBuildCacheConfiguration } = buildCacheConfig as IBuildCache;
          return new MyBuildCacheProviderModule.MyBuildCacheProvider({
           ...myBuildCacheConfiguration
          });
        }
      );
    });
  }
}

User Part:

setup in Repo

// rush-plugins.json
{
  "plugins": [
    {
      "packageName": "@scope/rush-custom-build-cache-plugin",
      "pluginName": "rush-custom-build-cache-plugin",
      "autoinstallerName": "plugins",
      // "optionsJsonFilePath": "some.json"
    },
}

// common/config/rush/build-cache.json
"cacheProvider": "my-custom-build-cache"

Use Case 2: custom command-line

Plugin Part:

// rush-plugin-manifest
{
  "plugins": [
    {
      "pluginName": "echo command",
      "description": "echo hello world",
      "commandLineJsonFilePath": "command-line.json"
    }
  ]
}

// command-line.json in plugin
{
  "commands": [
    {
      "name": "echo",
      "commandKind": "global",
      "summary": "Echo hello world",

      "shellCommand": "echo 'hello world'",

      "safeForSimultaneousRushProcesses": true
    }
  ]
}

User Part:

// rush-plugins.json
{
  "plugins": [
    {
      "packageName": "rush-custom-command-line-plugin",
      "autoinstallerName": "plugins",
      "pluginName": "rush-custom-command-line-plugin"
    }
  ]
}

run rush echo, and terminal will logout hello world

supports package binary

Let's say we want to use plugin defines perttier command

install prettier-quick in the specified autoinstaller (plugins here)

// command-line.json in plugin
{
  "commands": [

    {
      "name": "prettier",
      "commandKind": "global",
      "summary": "Used by the pre-commit Git hook. This command invokes Prettier to reformat staged changes.",

      // This will invoke common/autoinstall/plugins/node_modules/.bin/pretty-quick
      "shellCommand": "pretty-quick --staged",

      "safeForSimultaneousRushProcesses": true
    }
  ],
}

supports js file in plugin

// command-line.json in plugin
{
  "commands": [

    {
      "name": "echo",
      "commandKind": "global",
      "summary": "Echo",

      // <packageFolder> will be expanded at runtime
      "shellCommand": "node <packageFolder>/index.js",

      "safeForSimultaneousRushProcesses": true
    }
  ],
}

// index.js
console.log('hello world');

Things might need discussions

• Shall we move build-cache provider related config to rush plugin options?
  • If so, Do we need to do a breaking change for current amazon-s3 and azure configuration properties?
  • Personally, It makes more sense if isCacheWriteAllowed is provided by Rush, instead of a option in CloudBuildCacheProvider?
• rush-lib/node-library module in each plugin required by API (such as rushSession.getModule('xxx'))?
• It might be trivial for current plugin option declaration. If declare a optionsJsonFilePath, user need to create a options JSON file in the right path.
  • declare options directly inside rush-plugins, and dynamically generate rush-plugins.schema.json?

[octogonz]

meanwhile, rush-plugins.json use this new schema and nested options schema works!

{
  "$schema": "<relative_path>/rush-plugins.schema.json",
  "plugins": [
    {
      // other properties
      "pluginName": "some",
      "options": {
        // vscode hint/autocomplete works
      }
    }
  ]
}

FYI I seem to remember reading somewhere that the JSON Schema specification only permits URLs in the $schema field, even though some editors accept relative file paths. (If all the important editors do, then maybe it doesn't matter.)

Agreed, for now plugin configuration need specify optionsJsonFilePath explicitly to allow collisions. It might kinda inconvenience to use, since user need create options files manually and follow the plugin system convention strictly.

Naming collisions don't seem like a significant concern. For example the NPM package names are unique, so maybe the filename simply includes the package name?

Having each plugin's settings in a separate files has some upsides. It makes it much easier to sync settings between repos, and also to update the /** documentation blocks during an upgrade. Inline options only seems attractive for plugins with a small number of very simple options.

[chengcyber]

Update 2021/10/25

After another call with @octogonz, I'd like to record a TODO list here:

• Move manifest files under the specified autoinstaller folder
• Plugins with same name can only be applied once, thus removing optionsJsonFilePath in plugin configuration.
• Treat amazon-s3, azure cloud build cache plugin as default plugins, ensure making NO breaking changes to end user after this PR merged
• Ensure only import type from @microsoft/rush-lib, and get other imports from rushSession
• Limit the scope of this PR, no more features should be added
  • Clean life cycle
  • Clean public API in rush-lib

[octogonz]

@iclanton FYI I'd like to tune up rush-lib.api.md a bit before we release this feature, to eliminate some of these warnings and better distinguish experimental APIs from production APIs. However PR #2900 has accumulated a lot of work already, so I've decided to merge it and use a clean PR for the tune up.