Some modules are missing when using webpack 2

[HyperBrain]

Problem

We tried to use Webpack 2 and the current master branch of the webpack plugin.
The bundle generation is creating a correct bundle, but the packaged node modules,
that are installed as sibling to the bundle by the plugin are missing some external modules.

I.e. although they are present in the local services' package.json, they do not appear in
the package.json that is temporarily created by the webpack plugin to install the needed
external modules.

This leads to "required module not found" errors, when executing the compiled code on AWS
after Serverless deployed it.

Root cause

The plugin parses the webpack stats to find external modules by matching the returned stats
with "external ".

See comment below: Webpack 2's dependency optimization sometimes removes first level dependencies but keeps the removed dep's second level dependencies.

[HyperBrain]

I'll have to investigate this further. The package.json created by the plugin in .webpack is different regarding the dependencies than the original package.json. I'll reopen this as soon as I have more qualified information at hand.
Seems to be more than just the npm install.

[HyperBrain]

Reopened the issue after deeper investigation. Changed the description and subject to match my findings and the problem.

[HyperBrain]

Further analysis:

The stats object returned by webpack 2 contains all external references marked with "external ", including the second level dependencies.
It happens that webpack optimizes the imports so, that a first level dependency does not get imported, but one of its second level dependencies does.

This happened for us with react-addons-css-transition-group.
The primary dependency got optimized away, but instead it only references react-addons-transition-group which is a second level dependency of the css transition group module.

The plugin grabs all exported dependencies but cannot npm install the react-addons-transition-group because only its parent is mentioned in our package.json. So the (needed) dependency is missing in the resulting node modules.

A solution would be, that the module lookup is improved, so that each dependency reported by webpack 2 will be looked up with a deep search, beginning from the main service package.json throughout all of the (production) dependencies' dependencies. This would make sure that webpack 2's optimization will work with the plugin.

[HyperBrain]

It is even more complex. The new webpack 2 optimizer now emits partial modules as external references:

external "aws-sdk/global"
external "codemirror/addon/lint/javascript-lint"
external "codemirror/addon/lint/lint"
external "codemirror/addon/lint/yaml-lint"
external "codemirror/mode/javascript/javascript"
external "codemirror/mode/yaml/yaml"
external "crypto"
external "deep-assign"
external "hammerjs"
external "intl"
external "intl/locale-data/jsonp/en-US.js"
external "intl/locale-data/jsonp/pt-BR.js"
external "js-cookie"
external "js-yaml"
external "jshint"
external "markdown-to-jsx"
external "querystring"
external "react-codemirror"
external "react-desc"
external "react-dom/server"
external "react-dropzone"
external "react-facebook-login"
external "react-google-login"
external "react-html-document"
external "redux-thunk"
external "uuid/v4"

This means that the npm install approach will not work anymore, because the referenced externals might be only submodules or files out of an installed module.
The only way to handle this correctly is, in my opinion, copying all of the emitted externals right from the project's node_modules folder as webpack operated on exactly this.

[adambiggs]

Fixing #71 would also fix this it seems.

[Alex-Mann]

I am also experiencing a similar issue where I use yarn to install a local npm module (a library I have created) and webpack reports a "Webpack Missing Module 'Module Not Found'". Not sure if this is an error with webpack or this plugin as I didn't have time to dig too deeply

[daviskoh]

any updates on this?

[HyperBrain]

@daviskoh I stabilized the packaging in V3 and we run that already in our production environment. You can try it by using version ^3.0.0-rc.1. We did not experience "module losses" anymore.

However, the key to make it work well, is to use the new slsw.lib.entries automatic entry/handler detection and making sure that the function definitions in your serverless.yml point correctly to the handler files if they are located in subdirectories.