Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SINGLE_FILE option to embed all subresources into emitted JS #5296

Merged
merged 86 commits into from
Oct 16, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
5ea8765
Add SINGLE_FILE option to embed all subresources into emitted JS
buu700 Jun 13, 2017
3a562fa
misc cleanup as suggested by @kripken (#5296)
buu700 Jun 13, 2017
d2edd4c
replace --memory-init-file 0 logic with SINGLE_FILE-style embedding (…
buu700 Jun 13, 2017
e9b81cc
parse data URIs in JS to resolve cross-environment and CSP issues (#5…
buu700 Jun 14, 2017
cb300b6
minor cleanup (#5296)
buu700 Jun 14, 2017
2e33fb0
handle EMTERPRETIFY_FILE (#5296)
buu700 Jun 14, 2017
32bee1e
HTML worker fix (#5296)
buu700 Jun 15, 2017
a21e1d8
document SINGLE_FILE + HTML output (#5296)
buu700 Jun 15, 2017
e124141
include sodiumutil only when needed (#5296)
buu700 Jun 15, 2017
01b12b5
minor fixes (#5296)
buu700 Jun 15, 2017
0c82c16
binaryen get_subresource_location timing fix (#5296)
buu700 Jun 15, 2017
4fb1605
handle SINGLE_FILE in getBinary (#5296)
buu700 Jun 15, 2017
998f937
follow replacement string convention (#5296)
buu700 Jun 16, 2017
549a9b0
skip unneeded work for non-SINGLE_FILE compilation (#5296)
buu700 Jun 16, 2017
f7e889e
fix merge conflict (#5296)
buu700 Jun 17, 2017
9e5c34a
add myself to AUTHORS (#5296)
buu700 Jun 17, 2017
eeb547a
switch from sodiumutil to first-party base64 and UTF-8 implementations
buu700 Jun 17, 2017
b43f8ea
document encoding functions' behaviour (#5296)
buu700 Jun 18, 2017
c16ca3e
possible minor fix (#5296)
buu700 Jun 18, 2017
bc0074e
minor fix (#5296)
buu700 Jun 19, 2017
a21f18f
fix merge conflict (#5296)
buu700 Jun 19, 2017
19cbb02
getBinary minor cleanup (#5296)
buu700 Jun 19, 2017
7765795
merge bytesToString into intArrayToString (#5296)
buu700 Jun 19, 2017
73d1c91
handle data URI parsing in HTML output (#5296)
buu700 Jun 19, 2017
a6f46aa
move data URI parsing functions to preamble.js and rename for consist…
buu700 Jun 19, 2017
06ed6dc
factor out duplicated functions to arrayUtils.js (#5296)
buu700 Jun 21, 2017
8c74983
embed ASMJS_CODE_FILE only when needed (#5296)
buu700 Jun 26, 2017
0539459
embed mem init only when needed (#5296)
buu700 Jun 27, 2017
ec5cc8d
mem init method 0 fix (#5296)
buu700 Jun 27, 2017
962576f
handle ASSERTIONS in arrayUtils.js (#5296)
buu700 Jun 28, 2017
584b21c
html output condition comment (#5296)
buu700 Jun 29, 2017
362b79e
html output closure (#5296)
buu700 Jun 29, 2017
f0cb8da
do_binaryen SINGLE_FILE string replacement loop cleanup (#5296)
buu700 Jun 29, 2017
63e225a
arrayUtils maybeExport fix (#5296)
buu700 Jun 29, 2017
3711653
SINGLE_FILE regression tests (#5296)
buu700 Jun 29, 2017
dc1b69e
test fixes (#5296)
buu700 Jun 29, 2017
bd11a5e
SINGLE_FILE embed_memfile fix (#5296)
buu700 Jun 29, 2017
2e630bd
test cleanup (#5296)
buu700 Jun 30, 2017
5dece1f
arrayUtils.js embedding fix (#5296)
buu700 Jun 30, 2017
bd28f19
fix merge conflict (#5296)
buu700 Jul 5, 2017
ca3af49
node tryParseAsDataURI fix/cleanup (#5296)
buu700 Jul 5, 2017
638732b
base64 conversion cleanup (#5296)
buu700 Jul 5, 2017
5f93654
fix merge conflict (#5296)
buu700 Jul 6, 2017
200fb16
node intArrayFromBase64 fix (#5296)
buu700 Jul 10, 2017
fd28b2f
-g4 -> -g in SINGLE_FILE test to work around pre-exising bug (#5296)
buu700 Jul 10, 2017
ee64597
EMTERPRETIFY_FILE + SINGLE_FILE fixes (#5296)
buu700 Jul 10, 2017
6fa3a88
SINGLE_FILE + emterpreter test fix (#5296)
buu700 Jul 11, 2017
e3f7566
arrayUtils maybeExport cleanup (#5296)
buu700 Jul 11, 2017
e0f4e6c
SINGLE_FILE + EMTERPRETIFY_FILE test fix (#5296)
buu700 Jul 11, 2017
6b8a3e6
comment clarification (#5296)
buu700 Jul 13, 2017
67e41dc
SINGLE_FILE write in binary mode (#5296)
buu700 Jul 14, 2017
0f72d92
try XHR before tryParseAsDataURI where applicable (#5296)
buu700 Jul 14, 2017
97ae2ed
misc cleanup (#5296)
buu700 Jul 20, 2017
4452cb1
runtime size optimization (#5296)
buu700 Jul 20, 2017
3a2ccc4
INCLUDE_BASE64_UTILS -> SUPPORT_BASE64_EMBEDDING (#5296)
buu700 Jul 21, 2017
fa0ed7a
proxyClient SUPPORT_BASE64_EMBEDDING fix (#5296)
buu700 Jul 21, 2017
84c9e57
use regex to remove preprocessor directives from HTML output (#5296)
buu700 Aug 1, 2017
34d7e03
intArrayFromBase64 fixes for older Node and SpiderMonkey (#5296)
buu700 Aug 1, 2017
ebb8d7c
misc base64Utils fixes (#5296)
buu700 Aug 7, 2017
32f96f2
limited HTML output preprocessor support (#5296)
buu700 Aug 7, 2017
24e4356
run preprocessor on arrayUtils/base64Utils in jsifier (#5296)
buu700 Aug 7, 2017
b422419
misc touch ups (#5296)
buu700 Aug 7, 2017
1104ee0
clean up global var embed_memfile
kripken Aug 21, 2017
8814a69
clean up preprocessing
kripken Aug 21, 2017
bc33edd
use interpret-binary in other.test_single_file, so we don't depend on…
kripken Aug 21, 2017
d190392
minor cleanup (#5296)
buu700 Aug 21, 2017
86c319b
minor fixes (#5296)
buu700 Aug 22, 2017
62d9306
Merge remote-tracking branch 'upstream/incoming' into incoming
buu700 Sep 22, 2017
d55bc45
remove SINGLE_FILE dependency on Python preprocessor method (#5296)
buu700 Sep 26, 2017
a4a864d
minor fixes (#5296)
buu700 Sep 29, 2017
adbb60d
for better run-time efficiency, temporarily duplicate function pendin…
buu700 Sep 29, 2017
2bce73f
minor fix (#5296)
buu700 Sep 30, 2017
7489546
Merge remote-tracking branch 'upstream/incoming' into incoming
buu700 Sep 30, 2017
7e168bb
skip wasm streaming for SINGLE_FILE (#5296)
buu700 Sep 30, 2017
6fae984
s/request.response/response/ (#5296)
buu700 Oct 2, 2017
eb1fd57
Merge remote-tracking branch 'upstream/incoming' into incoming
buu700 Oct 5, 2017
8e4be3e
test update (#5296)
buu700 Oct 5, 2017
f33363b
proxyClient temporary fix pending Python preprocessor support (#5296)
buu700 Oct 8, 2017
50e58bd
SINGLE_FILE + Closure test case (#5296)
buu700 Oct 10, 2017
60fc52b
remove unneeded function (#5296)
buu700 Oct 10, 2017
eae3ed5
print arrayUtils/base64Utils earlier (via @erikdubbelboer) (#5296)
buu700 Oct 10, 2017
e8fd9e9
regression test for atob fallback issue (#5296)
buu700 Oct 10, 2017
8685abf
Merge remote-tracking branch 'upstream/incoming' into incoming
buu700 Oct 11, 2017
f98d511
HTML generation fix (#5296)
buu700 Oct 12, 2017
42992ef
atob function scope fix (#5296)
buu700 Oct 12, 2017
8e45599
asm.js synchronous init fix (#5296)
buu700 Oct 13, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 21 additions & 11 deletions emcc.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
if __name__ == '__main__':
ToolchainProfiler.record_process_start()

import os, sys, shutil, tempfile, subprocess, shlex, time, re, logging, urllib
import os, sys, shutil, tempfile, subprocess, shlex, time, re, logging, urllib, base64
from subprocess import PIPE
from tools import shared, jsrun, system_libs
from tools.shared import execute, suffix, unsuffixed, unsuffixed_basename, WINDOWS, safe_move
Expand Down Expand Up @@ -513,6 +513,16 @@ def filter_emscripten_options(argv):

# ---------------- Utilities ---------------

# Returns the subresource location for run-time access
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about moving this to tools/shared.py, perhaps a static method on the JS class?

def get_subresource_location(path):
if shared.Settings.SINGLE_FILE:
f = open(path, 'rb')
data = base64.b64encode(f.read())
f.close()
return 'data:application/octet-stream;base64,' + data
else:
return os.path.basename(path)

seen_names = {}
def uniquename(name):
if name not in seen_names:
Expand Down Expand Up @@ -799,7 +809,7 @@ def detect_fixed_language_mode(args):
options.separate_asm = True
logging.warning('forcing separate asm output (--separate-asm), because -s PRECISE_F32=2 or -s USE_PTHREADS=2 was passed.')
if options.separate_asm:
shared.Settings.SEPARATE_ASM = os.path.basename(asm_target)
shared.Settings.SEPARATE_ASM = get_subresource_location(asm_target)

if 'EMCC_STRICT' in os.environ:
shared.Settings.STRICT = os.environ.get('EMCC_STRICT') != '0'
Expand Down Expand Up @@ -1116,9 +1126,9 @@ def check(input_file):

if shared.Settings.BINARYEN:
# set file locations, so that JS glue can find what it needs
shared.Settings.WASM_TEXT_FILE = os.path.basename(wasm_text_target)
shared.Settings.WASM_BINARY_FILE = os.path.basename(wasm_binary_target)
shared.Settings.ASMJS_CODE_FILE = os.path.basename(asm_target)
shared.Settings.WASM_TEXT_FILE = get_subresource_location(wasm_text_target)
Copy link
Contributor Author

@buu700 buu700 Jun 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just went ahead and tested my change. (I was expecting a complicated and/or long setup process based on past experience installing from source with the SDK, but in retrospect I think it was just compiling clang that would take a long time.)

Anyway, it fails right here, apparently because the wasm, wast, and asm.js don't exist yet. How should I handle this?

shared.Settings.WASM_BINARY_FILE = get_subresource_location(wasm_binary_target)
shared.Settings.ASMJS_CODE_FILE = get_subresource_location(asm_target)

shared.Settings.ASM_JS = 2 # when targeting wasm, we use a wasm Memory, but that is not compatible with asm.js opts
shared.Settings.GLOBAL_BASE = 1024 # leave some room for mapping global vars
Expand Down Expand Up @@ -1624,12 +1634,12 @@ def repl(m):
# Copy into temp dir as well, so can be run there too
shared.safe_copy(memfile, os.path.join(shared.get_emscripten_temp_dir(), os.path.basename(memfile)))
if not shared.Settings.BINARYEN:
return 'memoryInitializer = "%s";' % os.path.basename(memfile)
return 'memoryInitializer = "%s";' % get_subresource_location(memfile)
else:
# with wasm, we may have the mem init file in the wasm binary already
return ('memoryInitializer = Module["wasmJSMethod"].indexOf("asmjs") >= 0 || '
'Module["wasmJSMethod"].indexOf("interpret-asm2wasm") >= 0 ? "%s" : null;'
% os.path.basename(memfile))
% get_subresource_location(memfile))
src = re.sub(shared.JS.memory_initializer_pattern, repl, open(final).read(), count=1)
open(final + '.mem.js', 'w').write(src)
final += '.mem.js'
Expand Down Expand Up @@ -2412,7 +2422,7 @@ def generate_html(target, options, js_target, target_basename,
meminitXHR.responseType = 'arraybuffer';
meminitXHR.send(null);
})();
''' % os.path.basename(memfile)) + script.inline
''' % get_subresource_location(memfile)) + script.inline

# Download .asm.js if --separate-asm was passed in an asm.js build, or if 'asmjs' is one
# of the wasm run methods.
Expand All @@ -2431,7 +2441,7 @@ def generate_html(target, options, js_target, target_basename,
}, 1); // delaying even 1ms is enough to allow compilation memory to be reclaimed
};
document.body.appendChild(script);
''' % (os.path.basename(asm_target), script.inline)
''' % (get_subresource_location(asm_target), script.inline)
else:
# may need to modify the asm code, load it as text, modify, and load asynchronously
script.inline = '''
Expand All @@ -2454,7 +2464,7 @@ def generate_html(target, options, js_target, target_basename,
document.body.appendChild(script);
};
codeXHR.send(null);
''' % (os.path.basename(asm_target), '\n'.join(asm_mods), script.inline)
''' % (get_subresource_location(asm_target), '\n'.join(asm_mods), script.inline)

if shared.Settings.BINARYEN and not shared.Settings.BINARYEN_ASYNC_COMPILATION:
# We need to load the wasm file before anything else, it has to be synchronously ready TODO: optimize
Expand All @@ -2468,7 +2478,7 @@ def generate_html(target, options, js_target, target_basename,
%s
};
wasmXHR.send(null);
''' % (os.path.basename(wasm_binary_target), script.inline)
''' % (get_subresource_location(wasm_binary_target), script.inline)

html = open(target, 'wb')
html_contents = shell.replace('{{{ SCRIPT }}}', script.replacement())
Expand Down
7 changes: 7 additions & 0 deletions src/settings.js
Original file line number Diff line number Diff line change
Expand Up @@ -856,6 +856,13 @@ var FETCH = 0; // If nonzero, enables emscripten_fetch API.

var ASMFS = 0; // If set to 1, uses the multithreaded filesystem that is implemented within the asm.js module, using emscripten_fetch. Implies -s FETCH=1.

var SINGLE_FILE = 0; // If set to 1, embeds all subresources in the emitted JS file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add examples of those files (data, asm.js, wasm etc.), i think it would help clarify things

// by converting their file names into base64 data URIs.
//
// Note that using this option may require a change to consuming
// pages' Content Security Policies -- specifically, adding data:
// to their connect-src directives.

var WASM_TEXT_FILE = ''; // name of the file containing wasm text, if relevant
var WASM_BINARY_FILE = ''; // name of the file containing wasm binary, if relevant
var ASMJS_CODE_FILE = ''; // name of the file containing asm.js, if relevant