[rush] plugin mode for customized cache provider

[m1heng]

Summary

as previously discussed in #2408 and along with more discussion made in Rush Hour. This issue intends to narrow the scope into solving customized cache provider along with a workable design of plugin mode.

cc @octogonz @chengcyber

[octogonz]

The first step is to work out a design spec. Compared to the problems of a generalized plugin system, the build cache providers have a somewhat easier problem because they are not needed until after rush install has completed successfully. So if it turns out to be complicated to get autoinstallers wired up to a plugin loader, we can likely skip that for issue #2898.

Some basic questions:

1. What does a Rush plugin NPM package look like? Probably it needs a special NPM package naming convention. A common question is whether a plugin == NPM package (1:1), or whether an NPM package can provide multiple plugins (1:many). For example: Heft's design is 1:many, so your NPM package name might be heft-example-plugin or heft-example-plugins depending on how many plugins are provided. Whereas ESLint's design is 1:1, -- the NPM package must be called eslint-plugin-example, and then individual features being provided are called "ESLint rules" not "plugins". I don't have a strong opinion about this, but I will insist that whatever terminology we choose should be very clearly specified (so it's easy to explain in docs like this). Hereafter I'll refer to them as "features" to avoid confusion between 1:1 vs 1:many.

2. What does the plugin manifest look like? The NPM package will have a "manifest" JSON file like my-rush-plugin/rush-plugin.json that Rush uses to query metadata about the features. For each feature, we need to know its name, its type, and maybe some technical properties like when it should be activated. For issue [rush] plugin mode for customized cache provider #2898 we can restrict our focus to one specific case (e.g. "featureKind": "buildCacheProvider")

3. How is the plugin registered with Rush? For example, maybe we have a common/config/rush/plugins.json config file that specifies the plugins.

4. When does Rush load the plugins? Above I have made an assumption that plugins are installed from an NPM registry. (That seems like the right design, but feedback is welcome.) We should start with the assumption that it gets installed using the autoinstaller engine. If that proves to be difficult, then maybe for the special "featureKind": "buildCacheProvider" case we would simply install it specially during rush install.

5. What does the basic plugin API look like? Since Rush is a heavyweight system, I feel like the integrations will be fairly shallow, and the API design may focus on protecting Rush from being destabilized by an unreliable plugin. This is rather different from Heft's scenario, where the plugins do most of the work, so we do very tight integration with Tapable. The api-documenter/src/plugin approach is maybe closer to what Rush needs. (?)

6. For build cache provider plugins, what does the API contract look like? In the current implementation it would be a subclass of CloudBuildCacheProviderBase. But this is probably too closely intertwined with Rush's internals. I'm not even sure that Terminal should be part of the contract -- maybe it should be a more primitive object such as ITerminalProvider. Whereas Heft plugins are likely to need to be recompiled and republished with each major Heft release, I feel like Rush plugins should be very robust, and if so will likely load a different @rushstack/node-core-library version from Rush itself.

[octogonz]

@m1heng For question 6, let us know if you can share any technical details about your CloudBuildCacheProviderBase subclass. For example, did it need to import any other modules from rush-lib? Did your fork need to modify any other Rush files besides the CloudBuildCacheProviderBase subclass?

[elliot-nelson]

Thanks for writing this up @octogonz! My two cents:

1. What does a Rush plugin NPM package look like?

Although right now we're focused on the cloud cache provider "feature", I'd assume we'll have multiple in the future. My vote would be for 1:many, so that a single NPM package could provide two or more complementary features (perhaps for a certain vendor, like GCP). I don't feel strongly about this either though. Either way a name like rush-xxxxxx-plugin seems to get the intent across.

2. What does the plugin manifest look like?

If you go with 1:many, the plugin manifest probably is an array of plugins, and each plugin might need to have its own schema. For example, even though the S3 and Azure cloud provider plugins are both plugins for the same feature, they can have different sets of required options that the user must provide, so we need some way for the plugin to specify its options schema, which Rush can validate (wherever the user "enables" the Azure plugin, they'd also need to provide the JSON options it expects, which Rush will validate on behalf of the plugin).

3. How is the plugin registered with Rush?

It's likely either common/config/rush/plugins.json or a section in rush.json. I don't have strong preference but I think I like the separate file more (it's got this unique thing going for it where it's trying to load custom schemas from each plugin for validation of provided options, potentially, which I think is safer as a totally separate operation from loading rush.json).

4. When does Rush load the plugins?

👍 Agree on pulling from an NPM registry (at least at first).

5. What does the basic plugin API look like?
6. For build cache provider plugins, what does the API contract look like?

My preference would be that a plugin implements a specific "feature interface" (like the cloud build cache provider), and receives its own validated options. As a strawman, maybe this looks something like:

import type { ICloudBuildCacheFeature } from '@microsoft/rush-lib';

export interface AzureBuildCacheProviderOptions {
    // ... options ...
}

export class AzureBuildCacheProvider implements ICloudBuildCacheFeature<AzureBuildCacheProviderOptions> {
    // constructor consumes options and prepares as necessary

    // implementation of the 2-3 methods ICloudBuildCacheFeature requires
}

export function register(....) {
    // some hook which you pass the AzureBuildCacheProvider object to register it
}

I don't think there is any "basic" API other than each one looks something like this (by that I mean, I'm not even sure there is an IFeatureBase; each feature is unique and you need to pick a specific one for a plugin, so I don't know there's a lot of required commonality between them).

As a potential plugin author, here are things I wouldn't want to deal with, and would definitely want Rush to help me with:

• Terminal output - I should be able to "act like Rush", spitting out informational, warning, and error messages, without having to do a bunch of wiring myself.
• Validating incoming options - I should be able to define my Options interface and a matching JSON schema, and trust that what I get is valid.
• Access to configuration - basic stuff like retrieving rush-project details, rush.json details, looking up rigged files, etc., would be a nice to have. (Although to your point, maybe we shouldn't do this through some passed-in context object... maybe the version of code for "looking up rigs" my plugin uses should be able to deviate from the user's actual rush-lib version.)

[chengcyber]

Hi @octogonz @iclanton , I drafted a graph below to show the plugin loader workflow as we talked eariler today. Can you help to check for the details and let me know if you have any concerns.

Details:

• a new rush command call rush update-plugins to update plugin manifests
• the manifest file store in common/rush-plugin-manifests/<package_name>/<package_version>/rush-plugin-manifest.json since there is a case where two plugins with same name while against different versions.
• we talked plugin options should go in individual files such as common/config/rush-plugins/<plugin_name>.json, but actually there might be a chance where user defines same plugins with different options, so i think plugin should explicitly declare which options.json should be used. e.g. optionsJsonFilePath property

PR related TODOs:

• refactor PluginManager to use new plugin load design
• rename RushSession to PluginContext
• Upgrade to Tapable@2
• provide registerCloudCacheProviderFactory
• validate plugin options by rush instead of plugin