Proposal: Bundling TS modules for consumption and execution

[weswigham]

Relates to #3089, #2743, and #17.

Goals

• Bundle js emit for TS projects while preserving internal modular structure (complete with ES6 semantics and despite being concatenated into a single file), but not exposing it publicly and wrapping the chosen entrypoint with a loader for the desired external module system.

Proposal

When all of --bundle, --module and --out are specified, the TS compiler should emit a single amalgamated js file as output. --bundle is a new compiler flag, indicating that the modules used should be concatenated into the output file and loaded with a microloader at runtime. The semantics of this microloader should preserve ES6 module semantics, but may not support TS-specific ES6 module semantics, such as import a = require or exports =. This is a major point of discussion, as to if these TS-specific syntax extensions are worth continuing to support with new language features. Additionally, this should wrap the result file with --module-specific declaration wrappers, if need be.

Supporting ES6 semantics is some effort - some rewriting/moving and declaration hoisting needs to be done to support circular references between internal modules.

Most of the effort involved here involves the design of the microloader and emit for said microloader - since this is compile time, and we don't want to register any new external dependencies at runtime (with this loader), I expect we can make some optimizations compared to, say, using an internal copy of systemjs.

Additionally, we need to decide if we want to make further optimizations based on the chosen --module - if the system chosen supports asynchronous module resolution (so not commonjs), then we can potentially use a more lazy, async dependency resolver internally, too.

So, to give an example of the emit (let's say we use a system-like thing internally and are emitting a system module):

Given:
a.ts:

export * from './b';
export * from './c';

b.ts:

export interface Foo {}

export class Bar {
    constructor() {
        console.log('');
    }

    do(): Foo { throw new Error('Not implemented.'); }
}

c.ts:

export class Baz {}
export interface Foo {}

Hypothetically, with --out appLib.js, --bundle a.ts and --module system arguments, we should get something like
appLib.js:

System.register([], function (exports_1) {
  var __system = /* loader code */;
  //a.ts
  __system.register('./a', ['./b', './c'], function(exports_1) {
    function exportStar_1(m) {
        var exports = {};
        for(var n in m) {
            if (n !== "default") exports[n] = m[n];
        }
        exports_1(exports);
    }
    return {
        setters:[
            function (_b_1) {
                exportStar_1(_b_1);
            },
            function (_c_1) {
                exportStar_1(_c_1);
            }],
        execute: function() {
        }
    }
  });
  //b.ts
  __system.register('./b', [], function(exports_1) {
    var Bar;
    return {
        setters:[],
        execute: function() {
            Bar = (function () {
                function Bar() {
                    console.log('');
                }
                Bar.prototype.do = function () { throw new Error('Not implemented.'); };
                return Bar;
            })();
            exports_1("Bar", Bar);
        }
    }
  });
  //c.ts
  __system.register('./c', [], function(exports_1) {
    var Baz;
    return {
        setters:[],
        execute: function() {
            Baz = (function () {
                function Baz() {
                }
                return Baz;
            })();
            exports_1("Baz", Baz);
        }
    }
  });

  function exportStar_1(m) {
      var exports = {};
      for(var n in m) {
          if (n !== "default") exports[n] = m[n];
      }
      exports_1(exports);
  }
  return {
    setters: [], 
    execute: function() {
      __system.import('./a').then(function (_pub_1) {
        exportStar_1(_pub_1);
      });
   };
});

[DanielRosenwasser]

What are the expected semantics of default exports? I believe we should error if we encounter multiple default exports when flattening into a bundle.

[weswigham]

A "default" export is literally an export named default. The keyword is just sugar for that (since default's a reserved word and all). It also has some special meaning with import statements... but that's less important here, given that the common point of conflict is when re-exporting things. Reexporting multiple defaults acts the same way exporting multiple Foo's would.

[mhegazy]

What about other module systems, i.e. amd/umd/commonjs would we use System as well? is there a better way to avoid perf penality on exporting an reexporting? how about using amd style IIFE?

Do you need a loader implementation? given that you know the right order, and dependency, the compiler can emit each module as an object, and pass the right objects to satisfy dependencies..

so something like:

define("a", ["require", "exports"], function (require, exports) {
    var _b = {};
    var _c = {};

    // c.ts
    (function (require, exports) {
        var Baz = (function () {
            function Baz() {
            }
            return Baz;
        })();
        exports.Baz = Baz;
    })(_c /* pass _c as exports*/);

    // b.ts
    (function(function (require, expots) {
        var Bar = (function () {
            function Bar() {
                console.log('');
            }
            Bar.prototype.do = function () { throw new Error('Not implemented.'); };
            return Bar;
        })();
        exports.Bar = Bar;
    })(require, _b /* pass _b as exports*/);


    // a.ts
    (function (require, exports, b_1, c_1) {
        function __export(m) {
            for (var p in m) if (!exports.hasOwnProperty(p)) exports[p] = m[p];
        }
        __export(b_1);
        __export(c_1);
    })(require, exports /*a is the entry module, so pass in the actual exports here*/,_b, _c);
});

[weswigham]

ES6 Modules can have circular dependencies, so we need a loader which can handle that (a la system). Simply ordering IIFEs isn't enough to retain ES6 semantics. If b and c have a bunch of mutually recursive functions (and they're allowed to), and b is ordered first, it may execute a function which calls into c whose definition isn't on the import object we've passed to b yet. The resolution to this is to hoist exported declarations out of the late execution block into an initialization area just like system.

I'm not saying we should use system internally (we shouldn't) but our loader/rewriter (probably more of a rewriter with helper functions) is going to do a lot of the same things. Plus we don't need to support async operations.

[mhegazy]

for that, you can do the same rewrite done today for loader, and execute all functions at the end. no need for a real loader though.

[kitsonk]

I might be wrong, but circular dependencies and IIFE debates are what have made it so we have ES6 modules, but no ES6 loader... 😉 The "rewriter" will have to do some sort of introspection of the modules to resolve circular dependencies to make sure you get the order and instantiation of the modules correct.

For reference, in AMD land, there is r.js, Dojo Builder and CramJS/RaveJS.

Also most of the AMD builders allow bundling of other resources as well (like text and JSON). I don't know if that is a consideration for this "bundler".

[weswigham]

@mhegazy We may still need some semblance of a loader if we want to support having a loader within the library at runtime so it can dynamically include dependencies - but it was removed in draft rev 28 and moved to an external spec so I'm not sure how much we support it?

We could have problems where users expect to be able to dynamically include their code with System and then not be able to if we actually use an internal loader (or no loader)... mmm... Maybe this is too much of a corner case to worry about? Or should it be supported? With commonjs modules dynamic loading was fairly common? To support this we'd actually have to teach the System (or internal module loader with all the semantics of a System loader) how to load the concatenated internal modules dynamically.

[mhegazy]

with commonjs modules dynamic loading was fairly common? To support this we'd actually have to teach the System (or internal module loader with all the semantics of a System loader) how to load the concatenated internal modules dynamically.

i am not sure i understand what you mean here.. can you elaborate.

[weswigham]

Something fairly common in node/commonjs scenarios is the requireDir function - a function which searches a directory for all JS files and require's them all (usually for side effects). This is a dynamic module load (since require isn't resolved statically, it could always be dynamic) - it is impossible for us to know what could be included at runtime in the non-static cases and inline references to the right dependency - we simply don't know what that dependency is yet. (In a browser, it's more likely that they're lazy-loading less needed modules for performance, rather than trying to blanket include a directory, but the dynamic nature of the import is still there.) In ES6 module-land, dynamic module loads are analogously done via System.import. (Versus the import keyword which is for static imports only.)

These dynamic loads pose a problem for bunding modules internally. What if we bundle the modules, but then the consumer of those modules loads them lazily via System.import rather than statically via import? We need to teach the available System how to import those internal modules.

[mhegazy]

but if i understand correctly, these all will be issues if we do provide a module loader implementation, if we do not, then these should just work. correct?

[weswigham]

No, if we do not then we'll have serious problems with people having modules in their bundle that they can't access in the way they'd expect to (in the above dynamic ways, anyway). It's why we'd need an internal loader/module lookup/whathaveyou.

[kitsonk]

Yes, lots of people have challenges understanding how CommonJS modules actually gets resolved. There is a lot of "magic". Many developers forget that when you move from the server side, where you have access to a file system and what is trivial resolution "magic" becomes wholly impossible on the client side. It is one of the many things which make people not like AMD, but I argue a lot of the "magic" we suffer to make lives easier for devs disadvantages the consumers of our solutions and actually we should bear a bit more of the burden.

I would be very cautious about introducing too much "magic" into module resolution. If there is to be any "magic" it is a configuration object which is utilised by the loader to map the final resolution. This of course can then be externalised our built in. Extending tsconfig.json might be the right way of expressing this.

[weswigham]

Just for clarification's sake, this is the essence of the troublesome dynamic case I'm considering:
main.ts:

import * from "ui";

//Draw UI
// ...
  instantiateLazyComponent("editor", "ComplexEditorElement").then(e => this.appendElement(e));
// ...

ui.ts:

export class MyElement {}

export function instantiateLazyComponent(componentType: string, className: string): Promise<MyElement> {
  return System.import(componentType+"_lazy.js").then(function(uiLazy: any) {
    if (!uiLazy[className]) throw new Error("Class not found");
    return new uiLazy[className]();
  });
}

editor_lazy.ts:

import {MyElement} from "ui"

export class ComplexEditorElement extends MyElement {}

graph_lazy.ts:

import {MyElement} from "ui"

export class ComplexGraphElement extends MyElement {}

This is (roughly, their injection framework's way more complex and they use script tags and global namespacing rather than modules) how lazy UI elements get loaded in the chrome devtools.

Supporting bundling those lazy components into the bundle is troublesome - we can put the code into the bundle, but we need to teach the System.import call how to find it, either by replacing it with out own internal loader (so we don't expose the internal modules), or by exposing those modules to the global loader.

We also can't eliminate the possibility that either editor_lazy or graph_lazy is loaded at runtime, despite them never being used in a static import.

[weswigham]

A reference for this feature: systemjs actually has a bundler. For the 'self-extracting' bundle, it adds roughly 440 lines of 'microloader' code to the bundle. It can bundle TS output. It doesn't handle dynamic imports in the bundle, though. (So you get imports which fail after bundling)

[SetTrend]

May I add my two cents (coming from #6419): Why is a call to System.import() done here anyway? Wouldn't that create a dependency to SystemJS?

I would think that ui.ts resembled something like:

export class MyElement {}

export function instantiateLazyComponent(componentType: string, className: string): Promise<MyElement>
{
  import component from "componentType+"_lazy.js";
  return component;
}

Then TSC can still decide whether the referenced component is included with the bundle (= no lazy loading necessary) or whether a --module specific loader may be invoked.

It would be necessary for editor_lazy.ts and graph_lazy.ts to export the same object names, of course, to be found by the same import.

[weswigham]

import component from componentType+"_lazy.js"; isn't valid ES6. In ES6, the spec says there is a module loader (as yet officially unspecified), and that module loader can handle dynamic imports such as this. (The ES6 spec just lists the semantics and syntax of static imports and modules in general) System isn't a reference to systemjs, but rather the name of the global the module loader is supposed to be on, according to the current proposed spec. systemjs is simply based of es6-module-loader which is meant to be a polyfill for this spec, so the terms appear conflated, as systemjs provides a System.

My original point was the regardless of what module system the user is compiling to, for completenesses' sake, we should provide an implementation of a canonical System which users can use for intra-bundle dynamic imports (whose registry isn't exposed to the external loader). My prototype implementation for this was actually commonjs based in structure, and the provided loader tried its best to follow commonjs semantics, but, naturally, since the source code is ES6, this leads to some expectations (like those around circular imports) getting neglected.

[hackwaly]

Webpack seems done this well.
https://github.com/webpack/webpack/tree/master/examples/harmony

[SetTrend]

Fully agree. As @mhegazy and @weswigham (and I) are argumenting pro - I'm curious: is there a roadmap for getting this implemented or is there perhaps a reason for not implementing it?

I'd like to add my personal point of view to:

import component from componentType+"_lazy.js"; isn't valid ES6.

TypeScript is not ES6. As long as there is no intention for having TypeScript sink into oblivion by becoming ES7 = TypeScript, I believe TypeScript should keep syntax peculiarities by its own.

I strongly believe that two different TypeScript constructs performing the same action should not be introduced. I'd prefer the team would either decide for using the import * from or the System.import() syntax - but not both of them with a non-congruent feature set.

[RyanCavanaugh]

This scenario seems well-served by many good external bundlers today; we don't need to rewrite esbuild