How to use this gem from within rails engines?

[fiedl]

I'm not sure if this is the right approach and maybe I've not understood this new part of the pipeline, yet, but:

Suppose, the major part of the app code lives inside a rails engine including all javascripts. And there are several main apps using that engine. The main apps only contain some layout changes and some minor patches.

Therefore, in order to integrate the webpacker gem into this setup, I'm trying to do the heavy lifting inside the engine, i.e. keep the work that has to be done inside all the main apps as little as possible.

How would I do that? Any pointers or suggestions are appreciated.

I'm not sure if this is a duplicate of #21. But in any case, I'd like to conclude this issue with a step-by-step guide how to approach this, for others facing the same use case.

What to do in the engine

1. Include webpacker in the *.gemspec file.
2. require 'webpacker' in the lib/foo/engine.rb.
3. ...

What to do in each main app

1. bundle install
2. Include ./bin/webpack-dev-server in the Procfile if using Foreman.
3. ...

[gauravtiwari]

@fiedl Out-of-the-box webpacker doesn't support engines yet. There are quite a few moving pieces that needs to be considered for this setup to work properly and usually people will have different use cases. I started doing some work on this but it's going to take time.

Feel free to leave any ideas/suggestions 👍

[fiedl]

@gauravtiwari Yes, after diving into this, I see your point. One would have to decide for each component where it belongs---to the engine, the main app, or both---and tie the pieces together in the right manner.

Some thoughts on this:

• The bin/webpack-dev-server is needed in the main app. One could add it to the Procfile of the main app to start the dev server with Foreman. Is there already a way to have the webpack dev server started by Pow?
• But one would need the ability to add a package from the engine.
• The app/javascript files have to be loaded from both, the engine and the main app. The main app files need to have the final say. I guess if there are several engines, the files from all engines need to be loaded.
• There are some files like .babelrc, .postcssrc.yml that would seem right in the main app root folder; but I'm not sure what these files are doing, yet.
• The config/webpack files appear to be tricky. In my use case, I would like to minimise the work to be done in the main app. But for other use cases, I imagine, the main app would need more control over the configuration. I really like how initializers work. I have most of them in the engine, but those of the main app have the last say in case of a conflict. Maybe we could utilise this mechanism: Would it make sense to tell webpacker in an initializer where to look for the webpack config? If no initializer is present, just use config/webpack from the rails root folder.
• For package.json and yarn.lock, one would need a mechanism similar to what is used with the *.gemspec, Gemfile and Gemfile.lock. If the engine decides that a certain package version is required, the package will be updated together with the engine when running bundle update my_engine from the main app. But the last say regarding the package versions has the main app as they are locked in the lock file. After all, the yarn.lock looks like a Gemfile.lock.
• This leads to the question how the upgrade process would look like from the view of the main app. I guess, just running bundle install from the main app would not cover any javascript package updates from the engine.

[himdel]

@gauravtiwari I started adding webpacker support to ManageIQ, where the UI is just an engine (so, app root != UI root), and we may need to support reading assets from multiple engines, and we need to output them to the rails root folder, not engine root.

To achieve that, I needed to override most of the webpacker methods to use the engine root instead of hardcoding Rails.root, I also needed to add a rake task that outputs Rails.root and call that from webpack config (didn't find another way of providing that path to webpack). (This also means that we need to include the initializer that overrides those paths from our Rakefile, which is a bit unfortunate :).)

.. And I still haven't figured out what would be needed to be able to call webpacker:compile, so far it it works if you manually call Webpacker.bootstrap but only when called from the root app, not when called from the engine.

I still haven't finished work on the "read assets from multiple engines" bit, but so far, I have a rake task that outputs a json of all the engines and their root paths (and filters those engines by existence of /app/javascript, until there's a proper way to register this), and intending to read that in the webpack config.

So.. I think at least these changes would be needed:

• don't assume all paths are relative to the current directory
• don't hardcode Rails.root anywhere
• come up with a sane way of passing those engine paths all the way to JS

If you're interested in the changes that we needed (and feel free to criticize and/or make suggestions), ManageIQ/manageiq-ui-classic#1132 .

---

Furthermore (and these are only my personal opinion):

• we should default to expecting the assets compiled, UI devels can enable webpack-dev-server, and for the rest, it should Just Work as long as they don't make UI changes.
• we should never deal with yarn, yarn and webpack are separate tools, and webpacker's job starts when you have all the assets, not before
• all those safety checks (compile depends on verify_install, verify_install depends on check_node and check_yarn, ..) are out of scope and needlessly complicate these tasks. All we need is to see the real error when it fails.

EDIT: Oh, and if there was a nice way of disabling the assets:precompile hook, it would be lovely, when wrapping that task, you may want the other taks to run on precompile :).

EDIT2: Aand making all the compile-time yarn deps as --dev, so that non-dev is reserved for actual UI dependencies, but I guess that's a separate issue :)

[zacksiri]

Right now i'm doing this, but obviously the solution is quite incomplete. Here is what I'm doing, what I see as the potential solution

by default wepbacker will always look for the manifest in the /public/packs folder of the Root application, that's because the default webpacker.yml is configured as such. However when coming from an engine environment it should respect the setting of the engine's webpacker.yml as well. which means webpacker needs to be aware of multiple namespaces.

For webpacker to be successful in an engine environment we need things to be isolated like every other part.

Which means, when we install the engine into an app, it needs to have an 'install step' that basically installs the pre-built modules into the rails app /public/packs/engine/name/manifest.json

We will probably need to modify the stylesheet_pack_tag and javascript_pack_tag helper to be aware of namespaces. So basically if you want to use the engine's assets you should be able to do something like javascript_pack_tag 'engine/name/vendor' etc... or javascript_pack_tag 'vendor', module: 'Engine::Name' which resolves to /public/packs/engine/name/vendor.js

I will have more clarity on this as I develop the solution. right now for me developing the engine with JS code I am just using a symlink from the dummy app's public/packs folder to the engine's public/packs folder, which works for development purposes.

A question for @dhh is also that should the Root app using the engine have to worry about compiling an engine's assets? If not it should make thing simpler since we would expect the JS code from the engine to already be compiled and usable which would mean we just need to make the javascript_pack_tag and stylesheet_pack_tag aware of those assets based on namespaces. or the module: 'Engine::Name' option that gets passed in.

[ohadlevy]

I've made a POC using webpack and rails engines that I believe that can be reused to solve this request: see ohadlevy/foreman@2afa796 for a reference.

[soundasleep]

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

[dhh]

Please Do Investigate 👍

…

On Jul 19, 2017, at 23:03, Jevon Wright ***@***.***> wrote: I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ohadlevy]

we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman project.
they key was to create a simple ruby script that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths

[dhh]

Sounds like something we should be able to automate 👍

…

On Thu, Jul 20, 2017 at 7:01 AM, Ohad Levy ***@***.***> wrote: we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman <https://github.com/theforeman/foreman> project. they key was to create a simple ruby script <https://github.com/theforeman/foreman/blob/develop/script/plugin_webpack_directories.rb> that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths <https://github.com/theforeman/foreman/blob/develop/config/webpack.config.js#L30> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#348 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKtcL8JHDO8sv-eFyRzPOOtBxPNcmoks5sP0GAgaJpZM4NUgTi> .

[mltsy]

Should the Root app using the engine have to worry about compiling an engine's assets?

On the surface, it's a fair argument that an asset engine (which is essentially what a webpack app gem is) could be expected to have all the assets pre-compiled, because... once it's released the assets aren't going to change... but looking at how assets in engines currently work, that is not the case, probably to allow including uncompiled assets within other uncompiled assets in the Root app (e.g. application.scss could use mixins from an engine's scss files) if that's the intent of the engine. I'm not totally sure whether that would ever be the case with a webpack engine I guess... is it feasible that there might be a use case for an engine that provides pieces to be used within a larger webpack app? Perhaps the engine would expose some Elm module or React component, and the Root app's UI would utilize that component from its own Elm/React app?

In Elm at least, that actually doesn't make sense as a use case, because all modules have to be explicitly included in the elm-package.json, and will then be added to the dependency graph independently of any Rails Engine. But I'm not sure the same can be said of every potential webpacked app - for instance, what if your Engine's app is just straight ES6 that exposes some modules? Could the Root app import those modules from the engine to be used as potential internal components of it's reactive UI?

If that kind of integration is possible (and useful, which seems likely to me if it's possible), then in order to facilitate it, a Root app obviously needs to be able to compile the engine's assets (after they are included and used however they are going to be used).

My main argument against allowing this type of integration, through a Rails Engine specifically, would be that it's mixing what are becoming ever more the separate responsibilities of the server-side app and the client-side app. That is, if your reactive UI requires some component, it should import it using a javascript dependency manager like npm, whereas the Gemfile should manage dependencies for your server-side Rails app (which may include some compiled reactive UI app to be used on the front-end, like any other asset to be sent to the client)

My question is: is that a good enough argument to make it impossible using webpacker (by assuming there will be no precompiled integration between the UI components of an engine and its root app, and hence not providing any considerations to facilitate such integration)?

[tomprats]

I was originally patiently waiting for this kind of support so I could upgrade a gem I manage to be used with webpacker, but @mltsy's argument resonated pretty well with me.

Javascript provided by an engine to be used through webpacker could just be put in (an) npm package(s). This would also allow the javascript to become part of the npm ecosystem, becoming a dependency or depending on other packages.

The engine could still provide CSS and JS when needed through sprockets. One downside is the potential overlap of an engine providing a page that depends on javascript that is also provided in its npm package. It could lead to a few use-cases of an end user having to download the same javascript code from multiple locations

[tomasc]

@mltsy @tomprats I partially agree with this view.

However, let's say the engine is a pluggable CMS that defines its own JS dependencies (similarly to gems in .gemspec). I would hope there would be a way for the main Rails app to pickup these dependencies as well.

As of now, in my case, most of the JS dependencies defined in a Rails engine are facilitated via https://rails-assets.org. It would be fantastic to be able to define them via webpacker.

[mltsy]

Yeah - this is a fairly unique situation where we have essentially two dependency managers in a single application (bundler and npm), and specifically the situation you're describing, where we have a Rails app with a bundler dependency on a Rails Engine, and an npm dependency on the "webpack app" (for lack of a better term, assuming it's not published as an npm package) that is defined in the Rails Engine. And that's the sticking point - the Rails engine "depends on" the webpack app that is implemented in its javascript/packs directory, but that dependency isn't really part of bundler or npm. So it's hard to say how/if it should be exposed in the contexts of npm and bundler. If we expose that somehow, we would essentially be adding javascript dependencies via the ruby dependency manager, which confuses things a bit when it comes to the javascript dependency manager... (there are some things included via an external channel, which creates somewhat of a mess when it comes to dependency management).

The best solution I can think of to handle that would be to somehow create/expose a kind of meta-package (or legitimate package?) out of the webpack app(s) contained within the Rails Engine, and automatically add that package to the packages.json of the Rails app before compiling with webpack (or maybe when the Engine is installed?). But this is all very theoretical, so maybe I should just shut my mouth and let someone who has attempted something like this share their wisdom ;)

[justin808]

@mltsy @tomasc @tomprats @ohadlevy @soundasleep @fiedl:

The beta version of React on Rails is built on top of Webpacker and here are the docs for using it.

I just released v9 beta.1, built on top of:

• Support React on Rails by merging Webpacker Lite #601
• VERSION 9.0: Update to Webpacker from Webpacker Lite shakacode/react_on_rails#908

[chimame]

How about specifying more than one in source_path of webpacker.yml?

default: &default
  source_path:
    - app/javascript
    - engine/javascript
  source_entry_path: packs
  public_output_path: packs
  cache_path: tmp/cache/webpacker

Specify the entry point of the application and the entry point of the engine, and build it with the application.
If it is dynamic, it will be glad even if it can be done dynamically with ERB etc.

[tomasc]

Thanks @chimame, looks like a good idea, I will give it a try and report back.

[lazylester]

@chimame I have successfully done something similar to your suggestion. Since all the engines in my app are in vendor/gems, my source_path definition in webpacker.yml is:

source_path: '**/app/javascript'

And this pulls in the modules from all the engines as well as the main app.

This is working very well for me, except I had to hack a single line in the rails webpacker npm module, as the manifest.json keys were not correct. I would happily create a PR for this, but I think there may be other use cases I'm not thinking of. Also, although my hack doesn't break the tests, I don't think this line of code is covered by tests and it's not clear to me why it was written that way.

I have a small example app with this implemented at https://github.com/lazylester/webpack_example

[mltsy]

Hmm... You're right! That's cool! At least partly... I mean that example app isn't quite the proof of concept that is necessary, because it doesn't actually require a module defined in the engine, but I bet it could!

I was slightly misunderstanding the role of npm and webpack. npm defines and downloads the sources (dependencies) from which webpack can choose what to include in any given pack. An engine is just another source for webpack to pull from, so it supplements the sources supplied by npm. Then webpack can choose from either npm's dependencies or the additional sources provided by the engine(s)...

Now... that is still a little bit disconcerting, just because those sources are outside of npm's dependency graph, meaning... the other thing we still need to solve is how to install the dependencies of the engine's pack(s). I see your "inventory" engine has no package.json (no JS dependencies) - that's not too likely in the real world, I imagine. Would you want to try adding and using one dependency and see if that works? I can't imagine running npm install on the main app would also install the engine's dependencies. But maybe there's a way to get around that... ? (I'm no expert in npm, so someone else here might have a better idea than I do about how to solve that issue)

[lazylester]

@mitsy you're right, I don't currently require any modules in the engine. I have done it elsewhere but forgot to include it in the example app. I'll update the example and post here. However my engines do not have any of their own npm dependencies, just local modules, within the engine, and a global (ractivejs) module that is in the main app's node modules.

So, yes, my approach has limitations. and doesn't cover many use cases. But it gets me to production!

[mltsy]

@justin808 - now that I understand what I'm looking for more I looked through the docs for using react_on_rails to see how you're handling this issue too. I see you're using a rails task to run a command provided by the engine (configuration.build_production_command) to compile the engine's pack, but (and this may be my ignorance about webpack) I see two issues in using this as a general solution:

1. It looks like that just runs webpack... when do the dependencies get installed? Does webpack install them? Does the user need to install them before running the task? (I see a yarn build:production version of the command in the test suite, but it looks like that script is not actually implemented or used)

2. If that command runs webpack with a totally separate webpack.yml, that wouldn't allow the parent application to include any of the engine's modules, etc. right? It's just compiling a totally independent pack? I see there's a comment just above the command saying if you don't want the engine to build the pack for you, that command can be set to nil, but I don't see any documentation on how to compile your pack in that case. Have you designed a workflow for compiling a webpack in the parent app that uses components/modules from your engine?

[mltsy]

Aha! Maybe we could use npm's local dependency feature?
https://docs.npmjs.com/files/package.json#local-paths

Yarn also respects this syntax... so we could tell the user to add this local dependency (file:vendor/gems/engine/? I'm not sure where the "package" root actually is - I would assume whatever directory the engine's package.json ends up in) to package.json in the parent app if they want to use any of the locally provided modules in their parent app's pack, and if not, provide a task for compiling the engine's pack independently! (could even make the installation part of an engine:install task)

I have to try this... when I get a chance... (if anybody else does, I'd love to hear the result)

[chimame]

@mltsy I think it's a good idea.

If both the idea to add to my webpacker.yml and the plan to add to your package.json come true, you can do the following.

• Write javascript_pack_tag 'engine_ javascript' in View (if it exists in webpacker.yml)
• import engine from 'engine_ javascript' can be defined in JavaScript (if it exists in package.json)

[lazylester]

@mitsy I updated my example app https://github.com/lazylester/webpack_example.

The engine now has a local module of its own and npm modules. You'd probably have to build the packs for the engines in addition to the packs for the main app in a build script, but I assume that's not a big deal, since there has to be a build script anyway, right?

I'll generate a PR for the webpacker change that permitted this to work. It may or may not be accepted, depending on whether it breaks something else. As I mentioned before, the tests still pass.

[lazylester]

Pull request: #875

[mltsy]

Nice! So... I guess there are two use-cases for engines here which have different solutions:

1. I want to use an Engine to provide uncompiled modules for use in a parent app's pack(s)
2. I want to use an Engine to provide one or more compilable packs to a parent app

In case 2, that's what react_on_rails seems to be doing - it provides a rails task for compiling the engine's pack, and then you can use it like any other asset. @lazylester and @chimame's solutions are nicer ways to be able to do that without having to run a separate task, by including the engine's entry points in the webpacker sources of the main app. ⭐

The use case I'm looking to solve for my personal use is case 1 though. And now that I think about it, we don't even need to include the engine's entry points in the webpacker sources to enable that, we just need to depend on it as a local dependency in package.json right? If that's true, it makes this all very clean, since there are orthogonal/independent solutions to each use case, and each can be employed depending on the purpose of the engine. (I still have to try the local dependency thing though for case 1, and the specifics of automating case 2 still need to be ironed out, since neither suggestion currently works with webpacker)

Since they seem to be orthogonal concerns, I suggest we open a new ticket for one of these use cases, and clarify the title of this ticket to represent the other - but I'm not sure which one corresponds best to the original post here... @fiedl ?

[jrochkind]

@tvdeyen are you choosing to release npm packages with same versions as your ruby gems, with every release, as rails does? If so, do you have any tooling to make that harder to mess up?

Do you have anything in place to ensure the user is using an appropriate or compatible version of the npm gem, corresponding to the ruby gem version they are using? Or if not "ensure", to even let them figure out what versions are compatible?

These are the two areas I could see leading to problems, and could really use some 'best practices' or examples or recommendations on.

[tvdeyen]

@jrochkind No, not yet. But I could imagine some kind of .alchemy-version.yml file (You will actually remember the famous spree_version.yml file aren't you 😆?

This is something that could probably live in Rails::Engine and be provided by Rails

@dhh how do you manage this in Rails gems and NPM modules? Is there something that could be make accessible for Engine authors? Would be great.

[tvdeyen]

@jrochkind something like this? 🤔

Edit: Version that uses the actual resolved npm package version

initializer "alchemy.check_package_version" do
  yarn_list = `yarn list --silent --flat --depth 0 --pattern alchemy_cms`
  package_version = yarn_list.match(/\d\.\d\.\d/).to_s
  if Gem::Version.new(package_version) < Alchemy.gem_version 
    abort "Your Alchemy npm package is outdated, please run `yarn upgrade @alchemy_cms/admin`"
  end
end

There are still a couple of caveats here:

1. It assumes you use a triple digit version number (The regex can easily be adopted to use 2 or 4 digits)
2. It is pretty slow to run the yarn list command in a ruby process. This adds a couple of milliseconds (~1000ms on my machine) to your app start up time. Maybe we only make this check in Rails.env.development?
3. Assumes your npm package version string is compatible with Gem::Version
4. Assumes that the released npm package has always the same version of the engine. Maybe the gem author holds a list of compatible npm package versions in a separate list and compares against it?

WDYT?

[jrochkind]

Yeah, I think this approach makes sense, but I suspect there will be all sorts of edge case details to get right, it'll have to be worked out. I believe for pre-releases, different version strings are required.

I think it would be nice if there were an out of the box maintained solution to this, perhaps from webpacker or rails. I think it's currently these issues are currently the biggest additional pain to separating JS into an npm package; with them solved it could seem no more challenging than the old sprockets-focused solution of packaging your JS in the gem.

[valterkraemer]

This is not perfect, but works pretty well.

It figures out what the paths are to the engines on the file system, and then adds them as entry points to the parent Webpacker.

Get paths to engines on file system

#! /usr/bin/env ruby
# file: get_engine_paths

require 'bundler'
require 'json'

# gem names in Gemfile
engine_names = ['my-gem-name']

engine_paths = Bundler.load.specs
  .select{ |dep| engine_names.include?(dep.name) }
  .map{ |dep| dep.to_spec.full_gem_path }

puts engine_paths.to_json

Needs to be executable (chmod +x get_engine_paths)

Make Webpacker find packs in engines

// file: webpack/environment.js
const { environment } = require("@rails/webpacker");

const { execSync } = require("child_process");
const { basename, resolve } = require("path");
const { readdirSync } = require("fs");

const babelLoader = environment.loaders.get("babel");

// Get paths to all engines' folders
const scriptPath = resolve(__dirname, "./get_engine_paths");
const buffer = execSync(scriptPath);
const enginePaths = JSON.parse(buffer);

enginePaths.forEach((path) => {
  const packsFolderPath = `${path}/app/javascript/packs`;
  const entryFiles = readdirSync(packsFolderPath);

  entryFiles.forEach((file) => {
    // File name without .js
    const name = basename(file, ".js");
    const entryPath = `${packsFolderPath}/${file}`;

    environment.entry.set(name, entryPath);
  });

  // Otherwise babel won't transpile the file
  babelLoader.include.push(`${path}/app/javascript`);
});

module.exports = environment;

Run yarn install in engines when yarn install is run in parent

Add to package.json "postinstall": "./engines_yarn_install.js"

#!/usr/bin/env node
// file: engines_yarn_install.js

const { spawn, execSync } = require("child_process");
const { resolve } = require("path");

// Get paths to all engines
const buffer = execSync(
  resolve(__dirname, "./get_engine_paths")
);
const enginePaths = JSON.parse(buffer);

enginePaths.forEach(enginePath => {
  spawn("yarn", ["install"], {
    env: process.env,
    cwd: enginePath,
    stdio: "inherit"
  });
});

[jrochkind]

@valterkraemer does that solution assume the path is always the same in every environment, or does it actually look it up "live"? At asset compile time? Asset delivery time?

Of course the path is usually diferent between a dev and production deploy, and sometimes can be different between different production deploys.

[valterkraemer]

It looks it up at asset build time, so it should have no problem with changing paths.

[tubbo]

@valterkraemer I have a similar approach, but it uses bundle info on the CLI to get a particular gem's information rather than relying on a pre-built JSON. It works pretty well on my own test app but I haven't actually done anything with it:

master...tubbo:add-installed-engines-to-additional-load-paths

You use it by calling the gem() function with the name of your gem, and adding to the module resolve paths in env config:

const { environment, gem } = require("@rails/webpacker")

environment.config.resolve.modules = [ gem("my-engine") ]

module.exports = environment

Then in your JS code:

import MyEngine from "my-engine"

This does require your code to be well namespaced, there's no automatic namespacing...it's just as if the files lived side by side in your app/javascript folder. So files in different gems with the same name would conflict with one another, and I suppose whichever file Webpack found first would be the one it used. But I think that's a decent trade-off for now.

[jrochkind]

@tubbo that's exciting. Do you think it would be feasible to turn it into a shareable package so other people can use this technique with share code? I guess it's functionality spans both JS and ruby, so maybe not?

[valterkraemer]

Thanks @tubbo. Weren't aware of bundle show, can now use that to get rid of the get_engine_paths file!

[brendon]

Not sure if this'll help anyone but I had good success with the following. In my setup I have a folder components within the root of my Rails application that then contains a bunch of engines.

1. In your engine, create a directory: app/javascripts/engine_name (e.g. components/blogs/app/javascripts/blogs
2. Within that directory you can do what you want. In my case I had a controllers directory. Within that I had the typical Stimulus.js loader (index.js):

// Load all the controllers within this directory and all subdirectories.
// Controller files must be named *_controller.js.

import { Application } from "stimulus"
import { definitionsFromContext } from "stimulus/webpack-helpers"

const application = Application.start()
const context = require.context(".", true, /_controller\.js$/)
application.load(definitionsFromContext(context))

A slight difference in this one is the require.context(".") instead of require.context("controllers") as otherwise this will look at the main app's controllers directory.

3. In config/webpacker.yml add a resolved_paths (or additional_paths if you use a newer version of webpacker).

resolved_paths: ['components/blogs/app/javascripts']

4. Now in your main app you can import your engine's controllers like so:

import 'controllers' // Should already be here
import 'blogs/controllers'

5. It's important to namespace your engine's javascript folder so that things don't collide.

Hopefully it's easy enough to see how this could be expanded for uses other than Stimulus.js.

[robyurkowski]

For anyone who's coming down this road, here's a summary of my research over the past few days:

There are four primary options in terms of using webpacker with a Rails Engine / Gem.

The first point of decision is whether you need your modules to be isolated from each other or if you're okay sharing between the host app and the engine. If you choose isolation, then your engine assets are compiled on their own and loaded separately. This is probably the right choice if you're worried about conflicts between the two sides, or if you can't realistically specify what version of a module the host app must use (e.g. a FOSS engine that has to operate as a black box and can't guarantee that the host app won't have a conflicting module version installed).

(If it's not quite clear what problem can occur here, imagine this situation: the engine's javascript is written using foo.js version 1, and it's installed into the host app which already uses foo.js version 2. Because there's not isolation, the engine uses whatever module import {} from "foo" loads, which could potentially (? -- I'm not a webpack expert by any means) be version 2, causing many errors because of an incompatible API.)

The problem is that isolation is tricky to achieve. There's no consensus among those who have attempted to install a separate instance of webpacker within their engine as to whether it works, let alone whether it works well. If you can force the host app to use the engine's module versions, i.e. you can say by dint of controlling both that they will both use foo.js version 1, then you should go that route because it's much easier.

Options 1 and 2: Shared Modules

What this should look like when it's all working:

// host: /app/javascript/packs/application.js

import Rails from "@rails/ujs"
//...

import "my_engine"

There are three primary obstacles to overcome with shared modules:

1. Get webpacker to load the engine's assets;
2. Get webpack-dev-server to watch the engine's assets and rebuild when they change;
3. Make sure that yarn install also triggers yarn install for engines (i.e. for ease of Heroku deployment).

In both cases, you'll need to set the engine up as follows. We'll assume that your engine is called my_engine.

1. cd /path/to/my_engine
2. mkdir -p app/javascript/my_engine;
3. Edit app/javascript/my_engine/index.js and insert:

// Require your modules here, but you'll run into conflicts if you 
// try to let the host app AND the engine start Rails-UJS / 
// Turbolinks etc... I generally let the host / dummy app start them. 
console.log("Called from my_engine.");

There are two different options here depending on whether the source for your engine lives within the host app.

Option One: Engine source within host app (e.g. /vendor/my_engine)

This option is largely cribbed from @brendon's answer above. Caveat lector that I haven't tried this option out, but from a theoretical basis it should work well. (Please let me know if it doesn't work).

Get webpacker to load the engine's assets

1. In config/webpacker.yml, change additional_paths to read additional_paths: ["/vendor/my_engine/app/javascript"]

Get webpack-dev-server to watch the engine's assets and rebuild when they change;

This is already taken care of by step 1.

Make sure that yarn install also triggers yarn install for engines

This particular answer is largely borrowed from @valterkraemer's answer, but I've simplified it to not need a separate script file at the cost of having to explicitly state which engines to yarn install. I'm also using an approach inspired by @tubbo but updated to use a non-deprecated command.

1. Edit the host application's package.json, and append the following before the final brace:

  "scripts": {
    "postinstall": "./scripts/yarn_postinstall.js"
  }

This tells yarn to execute scripts/yarn_postinstall.js after it finishes installing in the host app.

2. In the host app, create scripts/yarn_postinstall.js:

#!/usr/bin/env node

const { spawn, execSync } = require("child_process");
const { resolve } = require("path");

function yarn_install_engine(name) {
  const engineRoot = execSync(`bundle info --path ${name}`).toString().trim()

  spawn("yarn", ["install", "--ignore-scripts"], {
    env: process.env,
    cwd: engineRoot,
    stdio: "inherit"
  });
}

// Add your engines here.
yarn_install_engine("my_engine");

This script gets the path to the root of the engine and runs yarn install --ignore-scripts. You need the flag or this subsequent yarn install will trigger the post-install script we specified above, creating an endless loop.

3. chmod u+x scripts/yarn_postinstall.js.

You'll have to set the script executable or yarn will give you a permission error.

Option 2: Engine Source Not Vendored inside Host App

This will work whether you're using a local path for the engine or whether it's hosted somewhere else (i.e. you pull it down from Rubygems).

This is the solution that I wound up using and I can vouch for its completeness. I definitely need to credit @tubbo's answer here and need to praise his work on the gem() functionality, which should by all means be adopted into webpacker properly. In fact, this is basically a direct adaption of what he wrote but done within the host app, not webpacker proper.

Get webpacker to load the engine's assets

1. Within the host app's config/webpack/environment.js, right after const { environment} = require("@rails/webpacker"), insert:

const { execSync } = require('child_process')

function gem(name) {
  const root = execSync(`bundle info --path ${name}`).toString().trim()

  return `${root}/app/client`
}

environment.config.resolve.modules = [
  gem("my_engine"),
];

Get webpack-dev-server to watch the engine's assets and rebuild when they change;

1. Within the host app, create lib/webpacker/engine_extension.rb:

# frozen_string_literal: true

require "webpacker/configuration"

class Webpacker::Configuration
  def additional_paths
    fetch(:additional_paths) + resolved_paths + engine_paths
  end

  def engine_paths
    engines.map {|engine| "#{engine.root}/app/javascript" }
  end

  def engines
    defined?(::Rails) ? ObjectSpace.each_object(::Rails::Engine) : []
  end
end

When loaded after the engines, this looks up each Rails Engine and appends #{engine_root}/app/javascript to the additional_paths webpacker option.

2. Load the extension by editing the host's bin/webpack and bin/webpack-dev-server:

# Find the part that says...
Dir.chdir(APP_ROOT) do
  # And insert this next line:
  require "webpacker/engine_extension"

  Webpacker::WebpackRunner.run(ARGV) # or Webpacker::DevServerRunner.run(ARGV)
end

This loads the extension after the Rails engine has been loaded and required by bundler.

Make sure that yarn install also triggers yarn install for engines

Use the same answer as defined in Option 1 for this one :)

Isolated Modules

I include this section for completeness. This was the first thing that I tried and I certainly find this to be a much more challenging option. I wanted to be able to have my engine assets isolated from my host app, but I also wanted to make sure that I wasn't double-including heavy modules like Bootstrap, Tailwind, or even light things like Stimulus. That meant I needed to go the way of splitChunks and javascript_packs_with_chunks_tag, and I found that to be downright impossible to get working within a day or two. That's why I'd recommend you skip over option 3 and go to option 4 if this is the path that you're required to go.

Option 3: Install Webpacker in the Engine

Simply follow the guide in the 5-x-stable branch of this repository.

Option 4: Extract Assets to an NPM package

I did not go this route, but based on discussions in the thread above, particularly by @jrochkind, @tvdeyen, and @thebravoman, it seems like full isolation requires extracting your assets into a separate package for yarn. Prior art mentioned was the Ahoy project.

Wrap-up

Rails engines are absolutely critical pieces of functionality not just because they're the best way bar none to package up additional application-level functionality from the community, but also because they're excellent transitional structures that enable encapsulation of domain-differentiated functionality within an application. Their first-class support means that there's an easier path for a growing/groaning monolith to manage multiple bounded domains without having to either pollute the host app namespace or to move directly to some sort of service-oriented architecture. All the same, once there's appropriate first-class support for engines, then it's dead simple to move to SOA if necessary, since everything -- including assets -- is neatly packaged up.

I personally find their concept to be hugely enticing for managing bloat and technical debt. But until this point it's been an incomplete solution for me because of the missing asset piece. There's Sprockets, sure, but having to include Sprockets inside a webpacker-only host application complicates build steps and it sucks to go without the benefits of the modern build tooling.

I'm very happy that I managed to get this to work for me and I'm crossing my fingers that as I continue to build out my engine I don't discover some awful technical hurdle I've overlooked. And I hope that going forward there are simpler solutions for engines, as well as other first-class abstractions for maintaining strongly separated bounded domains within Rails apps. I wish I had the time to build it myself, but I'm hopeful that what I've compiled here can at minimum serve as a much more convenient and well-compiled jumping-off point for others that wind up with the same problem I have.

Finally, apologies for nickalerting anyone that didn't want it; I just wanted to make sure that I gave credit where it was due.

[jrochkind]

@robyurkowski wow, you seem to have figured out how to do some things I didn't think were possible! Thank you so much for that! I wonder if you want to write that up as a blog post somewhere too maybe? (dev.to?)

The option 2 seems closest in results to what we do/did with engines and sprockets -- that's what I had pretty much given up as not possible, but i just didn't understand webpack well enough to figure it out.(execSync... woah!)

I wonder if anything could/should be done in webpacker or another third party gem to encapsulate/DRY some of that, to make it even easier/more reliable to use? or in particular to avoid the need to "freedom patch" webpacker like this, add a feature to make it configurable instead of requiring a patch?

Although wait actually -- does this allow the engine to express it's own npm dependencies that will get integrated too, or no, just isolated assets? If not dependencies, that could be a barrier.

"first-class" support for an engine providing assets that can be referenced in the host app's webpacker would be a huge advance for rails webpacker ecosystem.

[justin808]

My gut tells me that the simplest way around the assets issues:

1. Use an NPM package for assets and specify that in the docs for installation.
2. Indicate any special asset handling rules in the docs as well if they are not defaults.

[robyurkowski]

I wanted to report back in that there was two other significant changes that I had to make with Option 2 above.

// host app: config/webpack/environment.js

environment.config.resolve.modules = [
  gem("my_engine"),
];

// Adding the line below
environment.loaders.get("babel").include.push(gem("my_engine"));

module.exports = environment;

Without this, the engine's NPM dependencies don't get picked up do get picked up but will crash the moment you try to load ES6 JS. There's probably more loaders that need to be patched but I'm not including any CSS in my engine yet so I haven't crossed that bridge.

The second thing is more of a resolution thing: instead of doing import controller from "my_controller", you'll need to make sure you namespace your assets lest you pick up the host app's. I often have to do import controller from "my_engine/controllers/my_controller". Likewise, require.context() needs to be properly scoped or you'll run into require issues.

(I'll try to encorporate these back into the body above when I get a bit of time.)

@jrochkind I'll definitely turn it into a blog post at some point -- thanks for the encouragement! With the above fixes, this does allow the engine to express its own NPM dependencies. I was able to get Stimulus, ActionCable, Cable_Ready, and StimulusReflex working almost purely in the engine.

There's a few things that need to be fixed up and that I need to clear up but I at least have a working proof of concept that you can put webpack assets in a Rails engine.

[jrochkind]

@robyurkowski I totally don't understand how engine's npm dependencies get picked up (and where engine would express them; just in a package.json at top-level of engine source dir?). I haven't spent time with it in code though, just reading your text.

Btu this sounds promissing. It's just complicated enough that, if it can be tested and proven... I wonder if it makes sense to create re-usable code packages to implement this? Maybe one npm package and one ruby gem, to turn incorporation of engine JS and dependencies into one line in JS and one line in ruby? But one step at a time. First blog post would be great, and the several of us trying to reproduce "manually" to verify it works for us, I guess?

[robyurkowski]

@jrochkind That's exactly right. I have a package.json that's in my engine root; there's also the yarn postinstall script in the host app that will trigger the engine's yarn install. The engine also produces its own node_modules folder and yarn.lock file but I've gitignored those.

And I agree with you -- this is complicated enough that even with my notes and git logs it's not super straightforward to build a sample app. I would definitely hope that we can get some of this merged into Webpacker and Rails proper. I'd rather not have to use external libs to do it if I can get my way; it's definitely not unusual that people want to load assets from their gems, let alone their engines. In fact, it's one of the critical things that's preventing the retirement of Sprockets, I think.

I'm under the gun right now but in the next few weeks I'll hopefully have some time to at least throw up a sample app and a blog post or two about my findings. In the meantime I think there's enough guidance in my post if you want to try and throw one together; I can definitely spare minutes here and there to debug if you run into problems.

[aharpervc]

in the next few weeks I'll hopefully have some time to at least throw up a sample app and a blog post or two about my findings

I think it'd be very helpful if you can post whatever you can here in this issue, so that anyone else that might be having this issue can pick up the torch.

I also definitely agree that this is a blocker to getting off of sprockets, for anyone who uses engines with assets, especially for private projects where making an NPM package isn't realistic

[justin808]

An open-source example is very helpful.

Check out what I did with https://github.com/shakacode/react_on_rails_tutorial_with_ssr_and_hmr_fast_refresh

[bablu-cs]

Hi, I'm trying to implement stripe using engine but can able access in the engine could you please help with this.

[brendon]

@bablu-cs, you'd be better asking on StackOverflow. This error isn't related to the issue we're talking about here as it's not related to webpacker. For some reason the root Stripe class isn't available to you so your application is trying to find it within your engine controller namespace. You could try using ::Stripe or make sure you're requiring the stripe code before trying to use it.