Module packaging

Background

For modules in Companion 3.0, we require them to be packaged with some special tooling. This is done to both reduce the number of files that your module spans on disk, and also the size. Loading code spread across hundreds of files is surprisingly slow on some oses, any by combining it all into a few files, we can often reduce the size from multiple mb, to a few hundred kb.
In previous versions this was handled by them being bundled inside of a giant zip file, but this has technical limitations and complexity that we do not wish to continue with.

Sometimes once built there are issues that prevent them from running, so it is a good idea to test it before distributing it. In our experience, issues often occur when working with files from disk, or introducing a new dependency that doesn't play nice. Once you have done this once, if you are just changing some internal logic you probably don't need to repeat this unless you want to be sure.

Packaging locally & Testing it

You can build your module into this format with yarn companion-module-build --dev. If this was successful, there should now be a pkg folder and a pkg.tgz file. These both contain the biult version of your module!
If you create an empty file DEBUG-PACKAGED in your module folder, then companion will read the code from pkg instead. This will let you test it.
You probably don't need to do a very thorough test, as long as it starts and connects to your device and a couple of actions work it should be fine.

If you encountered any issues in this process that you need help with, then have a look at the Module packaging wiki page for additional tips.

Distributing

When you run yarn companion-module-build, it produces a pkg.tgz. This file contains everything a user needs to be able to run your module. A tgz file is like a zip file, but different encoding. You can extract it into an appropriate folder and companion will be able to load it in.

In a future release of companion, we expect to add a way to import those files into companion so that users can get individual module updates. We don't want to rush this, so have decided to leave it out of the initial 3.0 release.

Customising the packaging

It is possible to adjust some settings in the packaging process, intended to help with some more complex modules.
Be careful when using these, as changing these are advanced features that can cause the build process to fail.

To start off, create a file build-config.cjs in your module folder, with the contents module.exports = {}. Each of these customisation sections will add some data to this file.

Changing __dirname

Webpack has a few options for how __dirname behaves in packaged code. By default it gets replaced with / which makes everything relative to the bundled main file. Alternatively, you can set useOriginalStructureDirname: true and the original paths will be preserved.

Including extra data files

Sometimes it can be useful to store some extra data as text files, or other formats on disk. Once your module is packaged, it wont have access to any files, from the repository unless they are javascript which gets included in the bundle or the files are explicitly copied. You will need to do this to allow any fs.readFile() or similar calls to work.

You can include these files by adding something like the following to your build-config.cjs:

module.exports = {
	extraFiles: [
        '*.txt',
    ],
}

You can use any glob pattern to define files to be copied.
All files will be copied to the root folder of the package, which is the same folder where the packaged main script is in. Make sure that there are no name conflicts when copying files from different folders.
Make sure you don't copy files you don't need, as these files will be included in the installation for all users of Companion.

Using native dependencies

Native dependencies are not possible to bundle in the same way as js ones. So to support these requires a bit of extra work on your part.

It is not yet possible to use all native dependencies. We only support ones who publish prebuilt binaries as part of their npm package.
This means that sharp and node-hid are not yet possible to use. Reach out if you are affected by this, we would appreciate some input. (node-hid has a PR to resolve this, and elsewhere in Companion we are using a fork base on this PR)

To support these modules, you should make one of two changes to your build-config.cjs, depending on how the library works.

If the library is using prebuild-install, then it will not work. With prebuild-install it is only possible to have the binaries for one platform installed, which isn't compatible with our bundling process. If you need one of these libraries, let us know and we can try and get this working.

If the library is using pkg-prebuilds for loading the prebuilt binaries, then you can use the following syntax.

module.exports = {
	prebuilds: [
		'@julusian/freetype2',
	],
}

If the library is using node-gyp-build, then there are a couple of options.
The preferred method is to set useOriginalStructureDirname: true in build-config.cjs. This changes the value of __dirname in your built module, and allows node-gyp-build to find its prebuilds.

If you are not able to use useOriginalStructureDirname: true, then you can instead mark the dependency as an external:

module.exports = {
	externals: [
		{
			'node-hid': 'commonjs node-hid',
		},
	],
}

This isn't the most efficient solution, as it still results in a lot of files on disk. We are looking into whether we can package them more efficiently, but are currently blocked on how most of these dependencies locate the native portion of themselves to load.

If the library is using something else, let us know which of these approaches works, and we can update this wiki to include it.

Using extra plugins

Sometimes the standard webpack functionality is not enough to produce working modules for the node runtime, but there is a webpack plugin which tackles your problem.

You can include additional plugins by adding something like the following to your build-config.cjs:

module.exports = {
  plugins: [
    new webpack.ProgressPlugin(),
  ],
}

Disabling minification

Some libraries can break when minified, if they are relying on names of objects in a way that webpack doesn't expect. This can lead to cryptic runtime errors.

You can disable minification (module-tools >=v1.4) with:

module.exports = {
	disableMinifier: true,
}

Alternatively, if you are having issues with error reports from users that have unreadable stack traces due to this minification, it can be disabled. We would prefer it to remain on for all modules to avoid bloating the install size (it can triple the size of a module), we do not mind modules enabling if it they have a reason to.

Using worker threads

Worker threads need their own entrypoints, and so need their own built file to execute.

We need to investigate how to handle this correctly. Reach out if you have ideas.

Common issues

This process can often introduce some unexpected issues, here are some of the more common ones and solutions:

TODO

Confusing log output

Due to how the packaging is done, it can result in some errors producing unreadable stack traces, or for a wall of code to be shown in the log making it unreadable. While using DEBUG-PACKAGED, if you run yarn companion-module-build --dev (the --dev parameter is key here) it will produce a larger build of your module that will retain original line numbers and formatting, making it much easier to read any output.