Rewrite JS transformer and scope hoisting in Rust

[devongovett]

This is a rewrite of the current JS transformer, including scope hoisting, in Rust. It utilizes SWC for parsing and codegen. All of the features of the existing JS transformer have been ported, including dependency detection, process.env replacement, fs inlining, node global insertion, and scope hoisting. In addition, SWC replaces Babel by default for transpilation when a browserslist is set, as well as for compiling JSX and TypeScript, and React Fast Refresh.

Overall, build times are up to 10x faster!

On the esbuild benchmark, without terser:

• Current v2: 34s
• This PR: 3.2s

With terser:

• Current v2: 56s
• This PR: 18s

Still plenty of optimization potential to go, but this is a great foundation to build on!

It's also really exciting that something like this is even possible. Parcel's language-agnostic core made this entirely possible in plugins. In fact, the old Babel-based plugins and scope hoisting implementation still work entirely without modification. In the future, we will be able to swap out additional plugins to improve performance or output quality as well.

Scope hoisting rewrite

Aside from performance, this PR was motivated by a need to rethink how Parcel's scope hoisting/tree shaking implementation works. We ran into some bugs especially having to do with preserving execution ordering (e.g. #5659) and duplicate execution that necessitated us to rethink how we handle wrapped assets via a module registry. The result is a hybrid between scope hoisting where we can, but falling back wrapping modules in a CJS-style function where needed. The packager is also redesigned to work entirely on strings rather than ASTs, which improves performance massively. You can read more about how this works in the design document for the new scope hoisting implementation.

• Fixes Module execution order is wrong with shared bundles #5659
• Fixes export default with a named function/class is still a live binding #5658
• Fixes {} is not a function #5657
• Fixes Cross-bundle $parcel$interopDefault execute wrapped assets on load #5662
• Fixes CJS modules should be wrapped in more cases #5606
• Fixes Non-static assignments to require are broken #5591
• Fixes exports is not defined at collection-strong.js #5892
• Fixes require is unavailable when scope hoisting enabled in Parcel 2 #5883
• Fixes TypeError: vB(...).default is not a function #5782
• Fixes CommonJS output format: export order #3989
• Fixes [Nightly.627] Maximum callstack error caused by naming clashes in built output #6051
• Fixes Transpilation differences between parcel .. and parcel build #6056
• Fixes Scope hoisting sometimes causes runtime error "$exports is not defined" #6071
• Fixes Javascript Math class is overridden by three.js module #6076
• Fixes Parcel 2 does assignment to undeclared variable when importing bundle text a certain way #6167

Possibly fixes (needs verification) #5064 #5914

Non scope hoisting issues: Fixes #5560

Migration

This only swaps out the current JS transformer and JS packager plugins in the default config. If your project has a browserslist and no .babelrc, your code will be transpiled with SWC instead of Babel. If you do have a .babelrc, nothing changes - Babel will still be used automatically. This means that both custom Babel plugins (e.g. CSS-in-JS transforms, Babel macros, etc.) continue to work out of the box. Scope hoisting, dependency collection, and everything else that was built into Parcel before will now occur in Rust, but this should be completely transparent.

This does open up the possibility to improve the performance of your build even further, however. You can now remove @babel/preset-env, @babel/preset-react, and @babel/preset-typescript from your .babelrc, and they will be automatically handled by SWC instead. This can significantly improve your build performance. If you have additional custom Babel plugins, you can leave them in your Babel config. If not, you can delete your Babel config entirely. We will likely add a warning in the future to assist with this migration.

Closes T-388, Closes T-194, Closes T-470, Closes T-849, Closes T-868

[devongovett]

We're actually using the SWC Rust crates and built a number of custom transforms in Rust on top of SWC. These are exposed to JS through a custom napi module, but we will also support wasm in the future (I believe @mischnic already has it working).

We're trying to keep as much as possible in Rust, but you're right, custom transform plugins are the next area to tackle. Ideally I'd love to be able to write custom transforms in Rust but this seems quite difficult considering Rust doesn't have a stable ABI. I hadn't considered using SWC as a parser for Babel but that's an interesting idea. Do you also convert the ast back to SWC after running babel and use SWC for codegen as well? Babel's code generator is really slow - we replaced with astring but it doesn't support all node types.

As for running Babel plugins directly in SWC, this sounds like a huge job with a large potential for bugs. The Babel API is massive, and there's tons of undocumented weird behavior in there. Are you planning on reimplementing this from scratch or just converting the AST to a Babel-compatible JS object and running Babel traverse itself? I imagine it would need to be written in JS anyway, so I'm not sure whether you'd really get much perf benefit from reimplementing vs just using the existing implementation.

the port of terser is WIP. The author of terser is porting it to rust using the system of swc.

Very excited about this. I'm planning on contributing to this effort as well when I have time. 😄

[kdy1]

Do you also convert the ast back to SWC after running babel and use SWC for codegen as well? Babel's code generator is really slow - we replaced with astring but it doesn't support all node types.

Currently not, but It's easy to implement. I didn't know that the code generator of babel is slow.
If it's slow, I think I need to implement conversion in the reverse direction anyway.

As for running Babel plugins directly in SWC, this sounds like a huge job with a large potential for bugs. The Babel API is massive, and there's tons of undocumented weird behavior in there. Are you planning on reimplementing this from scratch or just converting the AST to a Babel-compatible JS object and running Babel traverse itself? I imagine it would need to be written in JS anyway, so I'm not sure whether you'd really get much perf benefit from reimplementing vs just using the existing implementation.

I'll just use the system of babel. As it involves running js code, implementing the plugin system in rust would not boost performance.

Actually even making @swc/babel-plugin, which parses code, applies compatibility transforms, and gives the ast back to the babel is an option. Another option is to implement the babel plugin runner which uses packages from @babel/ just like babel.
I'm not sure which option is better.

[devongovett]

Makes sense. In regards to your earlier question:

Regarding babel plugin support, the current design only allows invoking babel plugins after applying transforms of swc. Is this enough?

I think it's fairly common to need custom plugins that operate on the source level rather than after compatibility transforms have been applied. For example, plugins might wish to do something with JSX elements (I know several CSS-in-JS plugins like emotion do this). So I think running plugins before SWC runs might actually be better. That's how we've currently set up Parcel's default JS pipeline - Babel runs first if there is a babel config, and after that we run SWC.

Actually even making @swc/babel-plugin, which parses code, applies compatibility transforms, and gives the ast back to the babel is an option. Another option is to implement the babel plugin runner which uses packages from @babel/ just like babel.

The first approach sounds easier, but is more limited. You wouldn't be able to replace the code generator easily, and I guess it might force SWC's transforms to run before other plugins. I think you'd want to parse using SWC, convert the AST to babel, run custom Babel plugins, convert AST back to SWC, run SWC transforms, codegen.

[devongovett]

Btw you might not need to implement the plugin runner from scratch if you invoke Babel programmatically by passing it an AST. You can also set an option to have it return an AST rather than codegen as well. This is how Parcel used to work as well.

[kdy1]

Thanks! I was also worried about the order of transforms.
Currently, reverse operation of .babelify is not implemented and it is the only restriction preventing invoking babel plugins before swc transforms. Thanks to your explanation, I noticed that implementing the reverse operation is the correct way to go :)

I'll implement babel ast to swc ast transformer. Do you want me to publish this crate on crates.io?

[devongovett]

Yeah we'd be happy to try it out when it's ready!

[mischnic]

(Apart from the worker count detection question)
And the failing macOS CI 😄

[zocky]

Is there a way to configure swc? AFAICT, .swcrc seems to be ignored.

[devongovett]

What are you looking to configure? At the moment, .swcrc is not used (the fact that Parcel uses SWC is a bit of an internal implementation detail). You can configure preset-env through targets or browserslist. And JSX and TypeScript should be handled automatically.

[zocky]

I'm using decorators (for mobx), my .swcrc would be

{
  "jsc": {
    "parser": {
      "syntax": "ecmascript",
      "decorators": true
    },
    "transform": {
      "legacyDecorator": true,
      "decoratorsBeforeExport": true
    }
  }
}