Allow parent plugins to declare what pluginOptions fields core should treat as subplugins

[pieh]

plugins field in pluginOptions is currently pretty special:

gatsby/packages/gatsby/src/bootstrap/load-plugins/load.js

Lines 142 to 150 in 9b2d51b

	// Plugins can have plugins.
	const subplugins = []
	if (plugin.options.plugins) {
	plugin.options.plugins.forEach(p => {
	subplugins.push(processPlugin(p))
	})
	plugin.options.plugins = subplugins
	}

We treat this as a way to define subplugins (mostly gatsby-transformer-remark being parent plugin and gatsby-remark-* being subplugins).

This works ok if there is single kind of plugins, or if different kind of subplugins have named exports so parent plugin know which to execute in various scenarios. The "issues" start happening when some plugin want to make use of plugins that were not created specifically for Gatsby. Example of this would be gatsby-plugin-mdx that makes use of 3 kinds of plugins (gatsbyRemarkPlugins, remarkPlugins and rehypePlugins) - because remark and rehype plugins are not created with Gatsby being only consumer, gatsby-plugin-mdx can't dictate contract between parent plugin and subplugins in this case (for example using named exports).

In this case gatsby-plugin-mdx needs to have 3 different list of 3 different kinds of subplugins, but right now they don't get special handling and so there are problems: traditional way of declaring plugins is to use plugin name that gatsby core resolves to absolute path and then parent plugin can require it (there are also issues there - see #21589). This path is skipped for gatsby-plugin-mdx and in cases where yarn/npm will not hoist subplugins (they don't end up in root node_modules and instead they are nested somewhere), then gatsby-plugin-mdx can't import them and that result in: Module not found errors that user can't recover from.

Example design of API

To solve the problem we need a way for gatsby-plugin-mdx to:
a) either declare that gatsbyRemarkPlugins, remarkPlugins and rehypePlugins should be treated as sub plugins (so core can resolve paths). For example:

// just examples (those are using top level pluginOptions field names)
// 1.
exports.registerSubpluginFields = () => [`gatsbyRemarkPlugins`, `remarkPlugins` and `rehypePlugins`]
// 2.
exports.onPreBootstrap: ({ actions }) => {
   actions.registerSubpluginFields( [`gatsbyRemarkPlugins`, `remarkPlugins` and `rehypePlugins`])
}

b) or provide helper utility to do that on arbitrary input (the helper function would need to be scoped to each plugin instance or to gatsby-config that declared group of plugins:

exports.onCreateNode = ({ resolvePlugin }, pluginOptions) => {
  // [...]
 
  pluginOptions.gatsbyRemarkPlugins.forEach(pluginDeclaration => {
    const resolvedPlugin = resolvePlugin(pluginDeclaration)
    const foo = require(resolvedPlugin.resolve)
  })
}

(note - just resolving the path will not fix issue with not running gatsby-x files APIs, so probably this is bad idea)
c) this could also be part of #16697, so the hypothetical onPluginOptions API isn't meant only for configuration merges, but would allow pluginOptions related tasks (including declaring subPlugins)

---edit:
On top of not being able to resolve module - we also are not running those subplugins gatsby-x files APIs (for example this remark subplugin implements gatsby-browser - https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-remark-images/src/gatsby-browser.js which either doesn't get used right now or gatsby-plugin-mdx need to have hacky support for by reimplementing core API runner).

[pieh]

added option c) that would encapsulate all pluginOptions related task in 1 API hook

[johno]

Related issues:

• [gatsby-plugin-mdx] create-webpack-config loader paths in pnpm context #23265
• [gatsby-plugin-mdx] cannot find module #21950

[codepunkt]

It would also be beneficial if gatsby-plugin-mdx and gatsby-transformer-remark supported the same plugins.