[metal-tools-soy] Add compilation cache for Soy files

[eduardolundgren]

I was wondering if make sense to cache compiled files to bump compilation performance.

One rough idea could be to write a cache file that contains the hashes of the files.

.cache

/a.soy 9e107d9d372bb6826bd81d3542a419d6
/b.soy e4d909c290d0fb1ca068ffaddf22cbd03
/c.soy 2bb8209c290d0fb1ca068ffaddf22cbd0

In the next build, before compiling all files, check the cache file and remove from the compilation file list the ones that the hash is still the same, then update the .cache file.

Dependencies

In case a.soy hasn't changed but it imports b.soy, if b.soy has changed a.soy also need to be recompiled. For that, we might use @mthadley tool to parse the files and calculate the dependencies.

Do you guys think it is doable?

[jbalsas]

Hey @eduardolundgren, that's definitely doable. We were actually discussing something along those lines for a prototype of a webpack loader that @p2kmgcl is working on...

One question, though, have you experienced a project where soy compilation is the limiting factor? From my observations, most of the build time goes to the transpilation phase.

We can easily instrument the tooling and start getting some answers.

[eduardolundgren]

@jbalsas On WeDeploy Console it's taking 10-15s to build the soy files, and most of the time only one file has been changed.

[jbalsas]

I'll take a look over the week, then. We can also tackle #9 (comment)

[robframpton]

Hey @eduardolundgren

I took a look into it, and it appears that the bundling is what's taking up most of that time. Which is consistent with what I've seen on Electric projects.

> [email protected] build /Users/rframpton/Projects/wedeploy/console
> magnet build -C config && gulp build

> info Config magnet.development.config.js
> info Building plugins…
buildSoy: 1768.503ms
buildClient: 10243.093ms
> info Building assets…
[10:25:31] Using gulpfile ~/Projects/wedeploy/console/gulpfile.js
[10:25:31] Starting 'marble'...
[10:25:31] Starting 'fonts'...
[10:25:31] Starting 'senna'...
[10:25:31] Finished 'senna' after 49 ms
[10:25:31] Finished 'fonts' after 77 ms
[10:25:31] Finished 'marble' after 283 ms
[10:25:31] Starting 'build'...
[10:25:31] Finished 'build' after 27 μs

From what I've seen in other webpack builds, one of the largest factors of increased build time is adding more entry points. It looks like the wedeploy console is currently passing 108 different entry points to webpack.

However, we definitely should still add a caching layer to the soy tools. It's just not going to address most the slowness you're experiencing.

[eduardolundgren]

Ugh, I didn't notice the build client was the most time-consuming.

buildSoy: 1768.503ms
buildClient: 10243.093ms

We can totally live with 1768.503ms for the soy compilation.

The focus now changes, would be cool to analyze on magnet-metal-plugin if there is any Webpack plugin that can handle the caching for file diff for us to save 10243.093ms. Do you know anything we can use to help this without requiring lots of changes on WeDeploy console?

[robframpton]

I think that integrating https://webpack.js.org/configuration/dev-server/ into magnet is going to be your best bet for improving the developer experience. They also have middleware that might be compatible with the magnet server.

[eduardolundgren]

@Robert-Frampton That sounds like a possibility, the only open question is how to plug soy compilation into the Webpack flow for this dev server. A good shot might be https://github.com/p2kmgcl/metal-soy-loader.