locateFile not used when loading SIDE_MODULES

[bdenhollander]

Experimenting with locateFile for browser cache invalidation and noticed that SIDE_MODULES specified as Module['dynamicLibraries'] = ['a.wasm', 'b.wasm'] do not pass through locateFile . The HTTP requests are missing ?v=revision. HTTP requests for app.wasm and app.data have the revision string appended as expected.

Module['locateFile'] = function (path, prefix) {
  if (path.endsWith('.wasm') || path.endsWith('.data')) {
    return prefix + path + '?v=' + window.REVISION;
  }
  return prefix + path;
}

Noticed in 2.0.16 and confirmed that it also occurs in 2.0.20. I attempted to test later versions but some of the recent compiler/linker flag changes are preventing the project from compiling.

Is locateFile an appropriate tool? Is this a bug in the loading of SIDE_MODULES?

[sbc100]

I guess there is argument to me made that the dynamic module loader should use locateFile too.

If you want to try a patch I think the place to modify would be in library_dylink.js:loadLibData. We probably want to call locateFile right before the following lines:

      if (flags.loadAsync) {                                                                                                            
        return fetchBinary(libFile);                                             
      }                                                                          
      // load the binary synchronously                                           
      return readBinary(libFile); 

[bdenhollander]

Adding libFile = locateFile(libFile); does the trick. Can this be merged into main?

I also found adding the revision to the dynamic library name as a temporary workaround.
Module['dynamicLibraries'] = ['a.wasm?v=revision', 'b.wasm?v=revision']

[sbc100]

BTW I'm not sure what you use case is but these days you can pass the side modules on the command line when linking your MAIN_MODULE and the dynamicLibraries will be populated automatically. In addition you can now use MAIN_MODULE=2 (if you pass the side modules on the command line at link time) and it will automatically includes just the transitive dependencies of all your modules (as opposed to MAIN_MODULE=1 which includes the entire JS and C system libraries).

[sbc100]

Yes, this change would be welcome, we would just need to open a PR and write a test for it.

[bdenhollander]

BTW I'm not sure what you use case is but these days you can pass the side modules on the command line when linking your MAIN_MODULE and the dynamicLibraries will be populated automatically. In addition you can now use MAIN_MODULE=2 (if you pass the side modules on the command line at link time) and it will automatically includes just the transitive dependencies of all your modules (as opposed to MAIN_MODULE=1 which includes the entire JS and C system libraries).

Thanks @sbc100. The project was using multiple libraries to allow for parallel downloads. I switched to packaging .so libraries inside app.data using --preload-file and --use-preload-plugins to avoid specifying dynamicLibraries.

Attempts to use MAIN_MODULE=2 across several versions between 2.0.16 - 2.0.24 have been unsuccessful thus far. I might open up a separate issue for that.

[sbc100]

There have been many improvement to MAIN_MODULE=2 if you are having issues with it still in 2.0.24, I'd love to know what they are. The only reason to use MAIN_MODULE=1 these days would be if you want to load dynamic libraries that are not known to you at the time of building your main module.

[bdenhollander]

Managed to get the project running in 2.0.24 with MAIN_MODULE=2 after making several changes:

• Added -Wno-unused-but-set-variable to ignore warnings not detected in 2.0.16
• Removed several compiler flags that are now linker only (passing flags for the side modules at the correct time in cmake was the most time consuming.)
• Added .so files to main module linker settings

	MAIN_MODULE=1	MAIN_MODULE=2
app.js	6381 KB	1414 KB
app.wasm	8228 KB	4428 KB

With the side modules passed in during linking, it doesn't want to load them via the file system from app.data and throws the following exception.

Uncaught (in promise) RuntimeError: abort(Assertion failed: need to see wasm magic number) at Error
    at jsStackTrace (app.js:2321)
    at stackTrace (app.js:2709)
    at abort (app.js:1232)
    at assert (app.js:690)
    at getDylinkMetadata (app.js:2282)
    at loadWebAssemblyModule (app.js:2459)
    at app.js:2639
    at abort (app.js:1234)
    at assert (app.js:690)
    at getDylinkMetadata (app.js:2282)
    at loadWebAssemblyModule (app.js:2459)
    at app.js:2639

They have to be available as individual files via HTTP and that bypasses locateFile.

[sbc100]

That (at least partly) great news. I'm keep to get all these issues addressed.

As a temporary way to get things working you could try using the new setting -sAUTOLOAD_DYLIBS=0. This is enabled by default and it means that any side module dependencies are implicitly added to dynamicLibraries. This is basically all it does:

emscripten/src/preamble.js

Lines 970 to 976 in 0883bcb

	#if MAIN_MODULE && AUTOLOAD_DYLIBS
	var metadata = getDylinkMetadata(module);
	if (metadata.neededDynlibs) {
	dynamicLibraries = metadata.neededDynlibs.concat(dynamicLibraries);
	}
	mergeLibSymbols(exports, 'main')
	#endif

. If you disable this then you will need to inject Module['dynamicLibraries'] yourself. If it is enabled (the default) you should not need to touch Module['dynamicLibraries'], it should just work.

I'm a little confused as to what that assertion is happening, and also what you mean by files via HTTP and that bypasses locateFile... i thought your patch added a call to locateFile in the case that loadWebAssemblyModule tries to load from HTTP. (IIUC we want to use locateFile only when the file is not present in the filesystem, is that right?)

[sbc100]

Oops, I just noticed a bug in that code I quoted above: #14524

[bdenhollander]

I'm a little confused as to what that assertion is happening, and also what you mean by files via HTTP and that bypasses locateFile... i thought your patch added a call to locateFile in the case that loadWebAssemblyModule tries to load from HTTP. (IIUC we want to use locateFile only when the file is not present in the filesystem, is that right?)

I've been through a few different combinations loading side modules.

Version 1 (2.0.16 + Module['dynamicLibraries']):
locateFile is skipped for .so 👎

Version 2 (2.0.16 + Module['dynamicLibraries'] + patch):
locateFile called for .so 👍

Version 3 (2.0.16 + bundle .so in app.data):
.so loaded from file system 👍
locateFile no longer needed for SIDE_MODULES

Version 4 (2.0.24 + bundle .so in app.data):
.so fails to load from file system 👎

emscripten/src/library_dylink.js

Lines 629 to 644 in 99942ab

	function loadLibData(libFile) {
	// for wasm, we can use fetch for async, but for fs mode we can only imitate it
	if (flags.fs) {
	var libData = flags.fs.readFile(libFile, {encoding: 'binary'});
	if (!(libData instanceof Uint8Array)) {
	libData = new Uint8Array(libData);
	}
	return flags.loadAsync ? Promise.resolve(libData) : libData;
	}
	if (flags.loadAsync) {
	return fetchBinary(libFile);
	}
	// load the binary synchronously
	return readBinary(libFile);
	}

When debugging, flags.fs is undefined so it's skipping loading from the file system. fetchBinary is retrieving an HTML page instead of the .so file. Attempting to interpret that as wasm is the source of Assertion failed: need to see wasm magic number.

[sbc100]

Thanks for the detailed explanation. So it sounds like we flags.fs is null when dynamic libraries are loaded during startup. We should look at addressing that perhaps.

I do have one question though: In version 3 above are the side modules loaded during startup? If so, how is flags.fs not also undefined in that case? (Or are you loading them after startup somehow?)

[bdenhollander]

As a temporary way to get things working you could try using the new setting -s AUTOLOAD_DYLIBS=0. This is enabled by default and it means that any side module dependencies are implicitly added to dynamicLibraries.

Adding this flag makes 2.0.24 function like 2.0.16. 👍

I do have one question though: In version 3 above are the side modules loaded during startup? If so, how is flags.fs not also undefined in that case? (Or are you loading them after startup somehow?)

2.0.24

• preloadDylibs attempts to load .so via URL
• fails to load dependencies and application does not start

2.0.16 and 2.0.24 with -s AUTOLOAD_DYLIBS=0

• --use-preload-plugins extracts .so from app.data and binary is passed to loadWebAssemblyModule
• dlopen retrieves .so via Module['preloadedWasm']
• loadLibData and flags.fs are not hit

[sbc100]

I see, so if we can make the auto-loading happen after the pre-loading you should be able to avoid AUTOLOAD_DYLIBS?

[sbc100]

Wait I'm still confused though.. are you using dlopen or does your main module depend on the libraries being loaded first?

If you are using dlopen in your application code then yes you probably want to use -sAUTOLOAD_DYLIBS=0 in any case.