Heroku slug size: can node_modules be deleted after build?

[nruth]

3 apps takes the slug size up towards 300MB, and it looks like most of this is in node_modules that shouldn't be necessary after the precompile step creates the javascript and css assets?

Poking around with heroku run bash and du is showing the node_modules directories as being around 200-500MB which will zip to some extent but shouldn't be necessary.

Is there a reason I've not thought of for keeping these around in the final dyno slug, or could we add something to delete the files from the slug once asset compilation succeeds?

There's something similar in the Heroku docs for Java.

[seanpdoyle]

@nruth thanks for bringing this to our attention.

Since Dynos and their filesystems are ephemeral, we cache node_modules and bower_components to speed up future builds.

If it's possible to remove directories from the slug, but keep them in the cache, that would be ideal.

Are you interested in investigating that possibility?

[nruth]

Yep that sounds good. I'm currently looking at a build-pack solution and will report back if I get it working. I've found one that removes files after other buildpacks and am trying it out on a staging remote.

[seanpdoyle]

@nruth is the simplest solution introducing a .slugignore file, or am I missing something?

[seanpdoyle]

Ah, I AM missing something:

However, it is not uncommon for there to exist files in the repo that are necessary for the build, but are not required at runtime.

--Lostmyname

[nruth]

Yep it seems like slugignore is more for excluding psd or pdf files from docs or artwork directories rather than cleaning up compilation artefacts. Probably needs a V2.

The buildpacks give another way to do this, but it's broken and took some poking around to get a working build.

When it worked, it reduced my slug size from ~296MB to 68MB by deleting the 3 ember project directories. The apps still seem to work after this, which I assume is because the compiled assets are stored somewhere in the rails app's asset pipeline instead of in those subdirectories?

My specific configuration is 3 ember apps as subdirectories (specifically, git submodules with a machine-user key). I've changed the names but it basically looks like this:

ember-cli-rails initializer:

EmberCli.configure do |c|
  c.app :'admin-panel'
  c.app :'member-panel'
  c.app :'reports-dashboard'
end

heroku buildpacks:add --index 3 https://github.com/moravcik/heroku-buildpack-post-build-clean.git#8f7caa624c0d0f9102a6cc6ec47f363a66ea0020

$ heroku buildpacks
=== app Buildpack URLs
1. heroku/nodejs
2. heroku/ruby
3. https://github.com/moravcik/heroku-buildpack-post-build-clean.git#8f7caa624c0d0f9102a6cc6ec47f363a66ea0020

.slug-post-clean containing:

admin-panel
member-panel
reports-dashboard

where each is a subdirectory containing the full ember-cli project and all the leftover node_modules that were causing the problem.

I don't know if dependency caching is working for my app so can't give a definite answer there, but the slug size is down and from what I understood of the buildpack docs the cache of the node or ruby buildpacks shouldn't be affected by another buildpack?

Long-term we're going to move to a redis lightning-deploy type setup because heroku push is getting really really slow now, but short-term I just wanted to get the slug size away from the 300MB red line so we can continue deploying and updating our project without fear of heroku rejecting the changes.

[nruth]

I'm not 100% confident this is correct yet, but going to continue with it through to production and see how it goes. It seems simpler than changing the gem.

What do you think of wrapping this up by me making a readme PR for the Heroku deployment instructions that briefly describes the problem and links to this issue and the buildpack?

[seanpdoyle]

What do you think of wrapping this up by me making a readme PR for the Heroku deployment instructions that briefly describes the problem and links to this issue and the buildpack?

Those gains are definitely worth mentioning in the README, maybe worth adding support to.

👍 Thanks for putting in the work!

[nruth]

Just to follow up on my previous hedging, we've been using this in production without issues since my original posts.

[thejchap]

Just an FYI for anyone else who stumbles on this...this doesn't work if you're using the include_ember_* helpers because ember-cli-rails-assets looks in those directories for a package.json (https://github.com/seanpdoyle/ember-cli-rails-assets/blob/master/lib/ember_cli/assets/paths.rb#L17).

Here's what my .slug-post-clean ended up looking like:

frontend/checkout/node_modules
frontend/checkout/bower_components
frontend/checkout/tmp

frontend/dashboard/node_modules
frontend/dashboard/bower_components
frontend/dashboard/tmp