Implements Promise-based API for modularized builds (#10697)

This removes the Module.then function and replaces it with a proper
Promise being returned from the initialization function.

That is, when in MODULARIZE mode, calling `Module()` does not
return an instance, but instead returns a Promise you can wait on
for when it is ready.

Fixes #5820

Author: Louis DeScioli

ChangeLog.md

@@ -24,6 +24,10 @@ Current Trunk
 
 1.39.15: 05/06/2020
 -------------------
+- Change the factory function created by using the `MODULARIZE` build option to
+  return a Promise instead of the module instance. If you use `MODULARIZE` you
+  will need to wait on the returned Promise, using `await` or its `then`
+  callback, to get the module instance (#10697).
 - Add `--extern-pre-js` and `--extern-post-js` emcc flags. Files provided there
   are prepended/appended to the final JavaScript output, *after* all other
   work has been done, including optimization, optional `MODULARIZE`-ation,

emcc.py

@@ -1320,9 +1320,6 @@ def has_c_source(args):
 
     if shared.Settings.MODULARIZE:
       assert not options.proxy_to_worker, '-s MODULARIZE=1 is not compatible with --proxy-to-worker (if you want to run in a worker with -s MODULARIZE=1, you likely want to do the worker side setup manually)'
-      # MODULARIZE's .then() method uses onRuntimeInitialized currently, so make sure
-      # it is expected to be used.
-      shared.Settings.INCOMING_MODULE_JS_API += ['onRuntimeInitialized']
 
     if shared.Settings.EMULATE_FUNCTION_POINTER_CASTS:
       shared.Settings.ALIASING_FUNCTION_POINTERS = 0
@@ -3404,22 +3401,17 @@ def modularize():
   logger.debug('Modularizing, assigning to var ' + shared.Settings.EXPORT_NAME)
   src = open(final).read()
 
-  # TODO: exports object generation for MINIMAL_RUNTIME. As a temp measure, multithreaded MINIMAL_RUNTIME builds export like regular
-  # runtime does, so that worker.js can see the JS module contents.
-  exports_object = '{}' if shared.Settings.MINIMAL_RUNTIME and not shared.Settings.USE_PTHREADS else shared.Settings.EXPORT_NAME
-
   src = '''
 function(%(EXPORT_NAME)s) {
   %(EXPORT_NAME)s = %(EXPORT_NAME)s || {};
 
 %(src)s
 
-  return %(exports_object)s
+  return %(EXPORT_NAME)s.ready;
 }
 ''' % {
     'EXPORT_NAME': shared.Settings.EXPORT_NAME,
     'src': src,
-    'exports_object': exports_object
   }
 
   if shared.Settings.MINIMAL_RUNTIME and not shared.Settings.USE_PTHREADS:

site/source/docs/getting_started/FAQ.rst

@@ -262,24 +262,13 @@ Here is an example of how to use it:
 
 The crucial thing is that ``Module`` exists, and has the property ``onRuntimeInitialized``, before the script containing emscripten output (``my_project.js`` in this example) is loaded.
 
-Another option is to use the ``MODULARIZE`` option, using ``-s MODULARIZE=1``. That will put all of the generated JavaScript in a function, which you can call to create an instance. The instance has a promise-like `.then()` method, so if you build with say ``-s MODULARIZE=1 -s 'EXPORT_NAME="MyCode"'`` (see details in settings.js), then you can do something like this:
+Another option is to use the ``MODULARIZE`` option, using ``-s MODULARIZE=1``. That puts all of the generated JavaScript into a factory function, which you can call to create an instance of your module. The factory function returns a Promise that resolves with the module instance. The promise is resolved once it's safe to call the compiled code, i.e. after the compiled code has been downloaded and instantiated. For example, if you build with ``-s MODULARIZE=1 -s 'EXPORT_NAME="createMyModule"'`` (see details in settings.js), then you can do this:
 
 ::
 
-    MyCode().then(function(Module) {
-      // this is reached when everything is ready, and you can call methods on Module
+    createMyModule().then((myModule) => {
+      // this is reached when everything is ready, and you can call methods on myModule
     });
-    
-.. note:: Be careful with using ``Module.then()`` inside ``Promise.resolve()`` or ``await`` statements. ``Module.then()`` resolves with ``Module`` which still has ``then``, so the ``await`` statement loops indefinitely. If you need to use ``Module.then()`` with ``await`` statement, you can use this workaround:
-
-    ::
-
-         await Module.then(m => {
-           delete m['then'];
-           resolve(m);
-         });
-
-    For more information read `this issue <https://github.com/emscripten-core/emscripten/pull/10697>`_.
 
 .. _faq-NO_EXIT_RUNTIME:
 

site/source/docs/porting/connecting_cpp_and_javascript/WebIDL-Binder.rst

@@ -113,28 +113,23 @@ Modular output
 
 When using the WebIDL binder, often what you are doing is creating a library. In that
 case, the `MODULARIZE` option makes sense to use. It wraps the entire JavaScript output
-in a function, which you call to create instances,
+in a function, and returns a Promise which resolves to the initialized Module instance.
 
 .. code-block:: javascript
 
-  var instance = Module();
-
-(You can use the `EXPORT_NAME` option to change `Module` to something else.) This is
-good practice for libraries, as then they don't include unnecessary things in the
-global scope (and in some cases you want to create more than one).
-
-Modularize also provides a promise-like API, which is the recommended way to use it,
-
-.. code-block:: javascript
-
-  var instance = Module().then(function(instance) {
-    // code that uses the instance here
+  var instance;
+  Module().then(module => {
+    instance = module;
   });
 
-The callback is called when it is safe to run compiled code, similar
-to the `onRuntimeInitialized` callback (i.e., it waits for all
-necessary async events). It receives the instance as a parameter,
-for convenience.
+The promise is resolved when it is safe to run compiled code, i.e. after it
+has been has been downloaded and instantiated. The promise is resolved at the
+same time the `onRuntimeInitialized` callback is invoked, so there's no need to
+use `onRuntimeInitialized` when using `MODULARIZE`.
+
+You can use the `EXPORT_NAME` option to change `Module` to something else. This is
+good practice for libraries, as then they don't include unnecessary things in the
+global scope, and in some cases you want to create more than one.
 
 
 Using C++ classes in JavaScript

src/closure-externs/closure-externs.js

@@ -999,7 +999,6 @@ var noExitRuntime;
 //    https://github.com/google/closure-compiler/issues/3167
 var BigInt;
 
-
 // Worklet
 /**
  * @constructor

src/postamble.js

@@ -68,7 +68,12 @@ if (memoryInitializer) {
     };
     var doBrowserLoad = function() {
       readAsync(memoryInitializer, applyMemoryInitializer, function() {
-        throw 'could not load memory initializer ' + memoryInitializer;
+        var e = new Error('could not load memory initializer ' + memoryInitializer);
+#if MODULARIZE
+          readyPromiseReject(e);
+#else
+          throw e;
+#endif
       });
     };
 #if SUPPORT_BASE64_EMBEDDING
@@ -121,34 +126,6 @@ if (memoryInitializer) {
 
 var calledRun;
 
-#if MODULARIZE
-// Modularize mode returns a function, which can be called to
-// create instances. The instances provide a then() method,
-// must like a Promise, that receives a callback. The callback
-// is called when the module is ready to run, with the module
-// as a parameter. (Like a Promise, it also returns the module
-// so you can use the output of .then(..)).
-Module['then'] = function(func) {
-  // We may already be ready to run code at this time. if
-  // so, just queue a call to the callback.
-  if (calledRun) {
-    func(Module);
-  } else {
-    // we are not ready to call then() yet. we must call it
-    // at the same time we would call onRuntimeInitialized.
-#if ASSERTIONS && !expectToReceiveOnModule('onRuntimeInitialized')
-    abort('.then() requires adding onRuntimeInitialized to INCOMING_MODULE_JS_API');
-#endif
-    var old = Module['onRuntimeInitialized'];
-    Module['onRuntimeInitialized'] = function() {
-      if (old) old();
-      func(Module);
-    };
-  }
-  return Module;
-};
-#endif
-
 /**
  * @constructor
  * @this {ExitStatus}
@@ -317,6 +294,9 @@ function run(args) {
 
     preMain();
 
+#if MODULARIZE
+    readyPromiseResolve(Module);
+#endif
 #if expectToReceiveOnModule('onRuntimeInitialized')
     if (Module['onRuntimeInitialized']) Module['onRuntimeInitialized']();
 #endif
@@ -427,9 +407,17 @@ function exit(status, implicit) {
     // if exit() was called, we may warn the user if the runtime isn't actually being shut down
     if (!implicit) {
 #if EXIT_RUNTIME == 0
-      err('program exited (with status: ' + status + '), but EXIT_RUNTIME is not set, so halting execution but not exiting the runtime or preventing further async execution (build with EXIT_RUNTIME=1, if you want a true shutdown)');
+      var msg = 'program exited (with status: ' + status + '), but EXIT_RUNTIME is not set, so halting execution but not exiting the runtime or preventing further async execution (build with EXIT_RUNTIME=1, if you want a true shutdown)';
+      err(msg);
+#if MODULARIZE
+      readyPromiseReject(msg);
+#endif // MODULARIZE
 #else
-      err('program exited (with status: ' + status + '), but noExitRuntime is set due to an async operation, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)');
+      var msg = 'program exited (with status: ' + status + '), but noExitRuntime is set due to an async operation, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)';
+      err(msg);
+#if MODULARIZE
+      readyPromiseReject(msg);
+#endif // MODULARIZE
 #endif // EXIT_RUNTIME
     }
 #endif // ASSERTIONS
@@ -482,15 +470,24 @@ if (!ENVIRONMENT_IS_PTHREAD) // EXIT_RUNTIME=0 only applies to default behavior
 #endif
 
 #if USE_PTHREADS
-if (!ENVIRONMENT_IS_PTHREAD) run();
+if (!ENVIRONMENT_IS_PTHREAD) {
+  run();
+} else {
 #if EMBIND
-else {  // Embind must initialize itself on all threads, as it generates support JS.
+  // Embind must initialize itself on all threads, as it generates support JS.
   Module['___embind_register_native_and_builtin_types']();
-}
 #endif // EMBIND
+#if MODULARIZE
+  // The promise resolve function typically gets called as part of the execution 
+  // of the Module `run`. The workers/pthreads don't execute `run` here, they
+  // call `run` in response to a message at a later time, so the creation
+  // promise can be resolved, marking the pthread-Module as initialized.
+  readyPromiseResolve(Module);
+#endif // MODULARIZE
+}
 #else
 run();
-#endif
+#endif // USE_PTHREADS
 
 #if BUILD_AS_WORKER
 

src/settings.js

@@ -1029,29 +1029,40 @@ var DETERMINISTIC = 0;
 
 // By default we emit all code in a straightforward way into the output
 // .js file. That means that if you load that in a script tag in a web
-// page, it will use the global scope. With MODULARIZE set, we will instead emit
+// page, it will use the global scope. With `MODULARIZE` set, we instead emit
+// the code wrapped in a function that returns a promise. The promise is
+// resolved with the module instance when it is safe to run the compiled code,
+// similar to the `onRuntimeInitialized` callback. You do not need to use the
+// `onRuntimeInitialized` callback when using `MODULARIZE`.
+// 
+// The default name of the function is `Module`, but can be changed using the
+// `EXPORT_NAME` option. We recommend renaming it to a more typical name for a
+// factory function, e.g. `createModule`.
 //
-//   var EXPORT_NAME = function(Module) {
-//     Module = Module || {};
-//     // .. all the emitted code from emscripten ..
-//     return Module;
-//   };
 //
-// where EXPORT_NAME is from the option of the same name (so, by default
-// it will be var Module = ..., and so you should change EXPORT_NAME if
-// you want more than one module in the same web page).
+// You use the factory function like so:
 //
-// You can then use this by something like
+//   const module = await EXPORT_NAME();
+//   
+// or:
 //
-//   var instance = EXPORT_NAME();
+//   let module;
+//   EXPORT_NAME().then(instance => {
+//     module = instance;
+//   });
+//   
 //
-// or
+// The factory function accepts 1 parameter, an object with default values for
+// the module instance:
 //
-//   var instance = EXPORT_NAME({ option: value, ... });
+//   const module = await EXPORT_NAME({ option: value, ... });
 //
 // Note the parentheses - we are calling EXPORT_NAME in order to instantiate
-// the module. (This allows, in particular, for you to create multiple
-// instantiations, etc.)
+// the module. This allows you to create multiple instances of the module.
+//
+// Note that in MODULARIZE mode we do *not* look for a global `Module` object
+// for default values. Default values must be passed as a parameter to the
+// factory function.
 //
 // The default .html shell file provided in MINIMAL_RUNTIME mode shows
 // an example to how the module is instantiated from within the html file.
@@ -1062,28 +1073,14 @@ var DETERMINISTIC = 0;
 // https://github.com/emscripten-core/emscripten/issues/7950)
 //
 // If you add --pre-js or --post-js files, they will be included inside
-// the module with the rest of the emitted code. That way, they can be
-// optimized together with it. (If you want something outside of the module,
-// that is, literally before or after all the code including the extra
-// MODULARIZE code, you can do that by modifying the JS yourself after
-// emscripten runs. While --pre-js and --post-js happen to do that in
-// non-modularize mode, their big feature is that they add code to be
-// optimized with the rest of the emitted code, allowing better dead code
-// elimination and minification.)
-//
-// Modularize also provides a promise-like API,
-//
-//   var instance = EXPORT_NAME().then(function(Module) { .. });
-//
-// The callback is called when it is safe to run compiled code, similar
-// to the onRuntimeInitialized callback (i.e., it waits for all
-// necessary async events). It receives the instance as a parameter,
-// for convenience.
+// the factory function with the rest of the emitted code in order to be
+// optimized together with it.
 //
-// Note that in MODULARIZE mode we do *not* look at the global `Module`
-// object, so if you define things there they will be ignored. The reason
-// is that you will be constructing the instances manually, and can
-// provide Module there, or something else, as you want.
+// If you want to include code outside all of the generated code, including the
+// factory function, you can use --extern-pre-js or --extern-post-js. While
+// --pre-js and --post-js happen to do that in non-MODULARIZE mode, their
+// intended usage is to add code that is optimized with the rest of the emitted
+// code, allowing better dead code elimination and minification.
 var MODULARIZE = 0;
 
 // If we separate out asm.js with the --separate-asm option,

src/shell.js

@@ -38,6 +38,15 @@ var Module = typeof {{{ EXPORT_NAME }}} !== 'undefined' ? {{{ EXPORT_NAME }}} :
 #endif // USE_CLOSURE_COMPILER
 #endif // SIDE_MODULE
 
+#if MODULARIZE
+// Set up the promise that indicates the Module is initialized
+var readyPromiseResolve, readyPromiseReject;
+Module['ready'] = new Promise(function(resolve, reject) {
+  readyPromiseResolve = resolve;
+  readyPromiseReject = reject;
+});
+#endif
+
 // --pre-jses are emitted after the Module integration code, so that they can
 // refer to Module (if they choose; they can also define Module)
 // {{PRE_JSES}}

src/shell_minimal.js

@@ -15,6 +15,15 @@ var Module = {{{ EXPORT_NAME }}};
 #endif // USE_CLOSURE_COMPILER
 #endif // SIDE_MODULE
 
+#if MODULARIZE
+// Set up the promise that indicates the Module is initialized
+var readyPromiseResolve, readyPromiseReject;
+Module['ready'] = new Promise(function(resolve, reject) {
+  readyPromiseResolve = resolve;
+  readyPromiseReject = reject;
+});
+#endif
+
 #if ENVIRONMENT_MAY_BE_NODE
 var ENVIRONMENT_IS_NODE = typeof process === 'object';
 #endif
@@ -131,6 +140,9 @@ function err(text) {
 // compilation is ready. In that callback, call the function run() to start
 // the program.
 function ready() {
+#if MODULARIZE
+  readyPromiseResolve(Module);
+#endif // MODULARIZE
 #if INVOKE_RUN && hasExportedFunction('_main')
 #if USE_PTHREADS
   if (!ENVIRONMENT_IS_PTHREAD) {