rustc: Add a `#[wasm_custom_section]` attribute #48883

alexcrichton · 2018-03-09T18:10:40Z

This commit is an implementation of adding custom sections to wasm artifacts in
rustc. The intention here is to expose the ability of the wasm binary format to
contain custom sections with arbitrary user-defined data. Currently neither our
version of LLVM nor LLD supports this so the implementation is currently custom
to rustc itself.

The implementation here is to attach a #[wasm_custom_section = "foo"]
attribute to any const which has a type like [u8; N]. Other types of
constants aren't supported yet but may be added one day! This should hopefully
be enough to get off the ground with some custom section support.

The current semantics are that any constant tagged with #[wasm_custom_section]
section will be appended to the corresponding section in the final output wasm
artifact (and this affects dependencies linked in as well, not just the final
crate). This means that whatever is interpreting the contents must be able to
interpret binary-concatenated sections (or each constant needs to be in its own
custom section).

To test this change the existing run-make test suite was moved to a
run-make-fulldeps folder and a new run-make test suite was added which
applies to all targets by default. This test suite currently only has one test
which only runs for the wasm target (using a node.js script to use WebAssembly
in JS to parse the wasm output).

rust-highfive · 2018-03-09T18:10:50Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

Mark-Simulacrum · 2018-03-09T18:26:35Z

src/tools/compiletest/src/runtest.rs

@@ -2359,11 +2359,6 @@ impl<'test> TestCx<'test> {
    }

    fn run_rmake_test(&self) {
-        // FIXME(#11094): we should fix these tests
-        if self.config.host != self.config.target {


hm, won't this make us run more tests now? Or do we handle this elsewhere for fulldeps suites somehow?

I think this was added long long ago when we still ran more tests in more situations (like tests on dist builders), but nowadays I don't think this is ever triggered. If I'm wrong though this'll just fail in CI

pepyakin · 2018-03-09T22:58:59Z

src/librustc/hir/check_attr.rs

+
+                    if attr.value_str().is_none() {
+                        self.tcx.sess.span_err(attr.span, "must be of the form \
+                            #[wasm_bindgen = \"foo\"]");


wasm_custom_section?

pepyakin · 2018-03-09T23:03:31Z

src/librustc_trans/back/wasm.rs

+pub fn add_custom_sections(path: &Path, sections: &BTreeMap<String, Vec<u8>>) {
+    let mut wasm = fs::read(path).expect("failed to read wasm output");
+
+    // see https://github.com/WebAssembly/design/blob/master/BinaryEncoding.md#module-structure


Can we really use https://webassembly.github.io/spec/core/binary/modules.html#custom-section for the reference?

bors · 2018-03-11T12:47:31Z

☔ The latest upstream changes (presumably #48799) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2018-03-14T15:44:20Z

☔ The latest upstream changes (presumably #48811) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · 2018-03-14T18:48:52Z

Travis is sad:

you need to replace ty::ParamEnv::empty(traits::Reveal::All) with ty::ParamEnv::reveal_all()

nikomatsakis

r=me once travis is happy

nikomatsakis · 2018-03-14T18:51:48Z

src/libsyntax/feature_gate.rs

@@ -455,6 +455,9 @@ declare_features! (

    // Parentheses in patterns
    (active, pattern_parentheses, "1.26.0", None, None),
+
+    // The #[wasm_custom_section] attribute
+    (active, wasm_custom_section, "1.26.0", None, None),


(don't we need a feature-gate test for this ?)

Heh probably!

alexcrichton · 2018-03-14T20:05:59Z

@bors: r=nikomatsakis

bors · 2018-03-14T20:05:59Z

📌 Commit 464f338 has been approved by nikomatsakis

alexcrichton · 2018-03-14T20:06:37Z

@bors: r=nikomatsakis

bors · 2018-03-14T20:06:38Z

📌 Commit 5d760d3 has been approved by nikomatsakis

alexcrichton · 2018-03-14T21:12:02Z

@bors: r=nikomatsakis

bors · 2018-03-14T21:12:02Z

📌 Commit 3c9dcad has been approved by nikomatsakis

bors · 2018-03-22T20:13:34Z

☔ The latest upstream changes (presumably #49264) made this pull request unmergeable. Please resolve the merge conflicts.

This commit is an implementation of adding custom sections to wasm artifacts in rustc. The intention here is to expose the ability of the wasm binary format to contain custom sections with arbitrary user-defined data. Currently neither our version of LLVM nor LLD supports this so the implementation is currently custom to rustc itself. The implementation here is to attach a `#[wasm_custom_section = "foo"]` attribute to any `const` which has a type like `[u8; N]`. Other types of constants aren't supported yet but may be added one day! This should hopefully be enough to get off the ground with *some* custom section support. The current semantics are that any constant tagged with `#[wasm_custom_section]` section will be *appended* to the corresponding section in the final output wasm artifact (and this affects dependencies linked in as well, not just the final crate). This means that whatever is interpreting the contents must be able to interpret binary-concatenated sections (or each constant needs to be in its own custom section). To test this change the existing `run-make` test suite was moved to a `run-make-fulldeps` folder and a new `run-make` test suite was added which applies to all targets by default. This test suite currently only has one test which only runs for the wasm target (using a node.js script to use `WebAssembly` in JS to parse the wasm output).

This commit adds a new attribute to the Rust compiler specific to the wasm target (and no other targets). The `#[wasm_import_module]` attribute is used to specify the module that a name is imported from, and is used like so: #[wasm_import_module = "./foo.js"] extern { fn some_js_function(); } Here the import of the symbol `some_js_function` is tagged with the `./foo.js` module in the wasm output file. Wasm-the-format includes two fields on all imports, a module and a field. The field is the symbol name (`some_js_function` above) and the module has historically unconditionally been `"env"`. I'm not sure if this `"env"` convention has asm.js or LLVM roots, but regardless we'd like the ability to configure it! The proposed ES module integration with wasm (aka a wasm module is "just another ES module") requires that the import module of wasm imports is interpreted as an ES module import, meaning that you'll need to encode paths, NPM packages, etc. As a result, we'll need this to be something other than `"env"`! Unfortunately neither our version of LLVM nor LLD supports custom import modules (aka anything not `"env"`). My hope is that by the time LLVM 7 is released both will have support, but in the meantime this commit adds some primitive encoding/decoding of wasm files to the compiler. This way rustc postprocesses the wasm module that LLVM emits to ensure it's got all the imports we'd like to have in it. Eventually I'd ideally like to unconditionally require this attribute to be placed on all `extern { ... }` blocks. For now though it seemed prudent to add it as an unstable attribute, so for now it's not required (as that'd force usage of a feature gate). Hopefully it doesn't take too long to "stabilize" this! cc rust-lang-nursery/rust-wasm#29

alexcrichton · 2018-03-22T20:17:07Z

@bors: r=nikomatsakis

bors · 2018-03-22T20:17:08Z

📌 Commit d889957 has been approved by nikomatsakis

…r=nikomatsakis rustc: Add a `#[wasm_custom_section]` attribute This commit is an implementation of adding custom sections to wasm artifacts in rustc. The intention here is to expose the ability of the wasm binary format to contain custom sections with arbitrary user-defined data. Currently neither our version of LLVM nor LLD supports this so the implementation is currently custom to rustc itself. The implementation here is to attach a `#[wasm_custom_section = "foo"]` attribute to any `const` which has a type like `[u8; N]`. Other types of constants aren't supported yet but may be added one day! This should hopefully be enough to get off the ground with *some* custom section support. The current semantics are that any constant tagged with `#[wasm_custom_section]` section will be *appended* to the corresponding section in the final output wasm artifact (and this affects dependencies linked in as well, not just the final crate). This means that whatever is interpreting the contents must be able to interpret binary-concatenated sections (or each constant needs to be in its own custom section). To test this change the existing `run-make` test suite was moved to a `run-make-fulldeps` folder and a new `run-make` test suite was added which applies to all targets by default. This test suite currently only has one test which only runs for the wasm target (using a node.js script to use `WebAssembly` in JS to parse the wasm output).

Rollup of 15 pull requests - Successful merges: #48265, #48528, #48552, #48624, #48883, #48909, #49028, #49030, #49102, #49160, #49169, #49203, #49262, #49272, #49295 - Failed merges: #48942, #49035

eddyb · 2018-03-28T19:49:56Z

Do we want to use #[section_name] for this long-term by any chance?

alexcrichton · 2018-03-28T23:02:45Z

@eddyb perhaps, although it has a pretty radically different meaning on wasm so far which doens't map well (aka we bypass llvm completely)

alexcrichton force-pushed the wasm-custom-sections branch from f2caef6 to 21fa7d4 Compare March 9, 2018 18:10

rust-highfive assigned nikomatsakis Mar 9, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 9, 2018

alexcrichton mentioned this pull request Mar 9, 2018

Support for emitting custom sections rustwasm/team#30

Closed

Mark-Simulacrum reviewed Mar 9, 2018

View reviewed changes

alexcrichton force-pushed the wasm-custom-sections branch from 21fa7d4 to 3f08d32 Compare March 9, 2018 19:39

pepyakin reviewed Mar 9, 2018

View reviewed changes

alexcrichton force-pushed the wasm-custom-sections branch from 3f08d32 to 46507a0 Compare March 9, 2018 23:17

alexcrichton force-pushed the wasm-custom-sections branch 2 times, most recently from 49a8df8 to a89b095 Compare March 12, 2018 14:04

alexcrichton mentioned this pull request Mar 13, 2018

rustc: Add a #[wasm_import_module] attribute #48992

Closed

alexcrichton force-pushed the wasm-custom-sections branch from a89b095 to ef2e7c3 Compare March 14, 2018 15:46

nikomatsakis approved these changes Mar 14, 2018

View reviewed changes

nikomatsakis added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 14, 2018

alexcrichton force-pushed the wasm-custom-sections branch from ef2e7c3 to 464f338 Compare March 14, 2018 20:05

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 14, 2018

alexcrichton force-pushed the wasm-custom-sections branch from 464f338 to 5d760d3 Compare March 14, 2018 20:06

alexcrichton force-pushed the wasm-custom-sections branch from 5d760d3 to 3c9dcad Compare March 14, 2018 21:11

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 22, 2018

alexcrichton added 2 commits March 22, 2018 13:16

alexcrichton force-pushed the wasm-custom-sections branch from 44ca018 to d889957 Compare March 22, 2018 20:17

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 22, 2018

alexcrichton mentioned this pull request Mar 23, 2018

Rollup of 15 pull requests #49308

Merged

alexcrichton merged commit d889957 into rust-lang:master Mar 23, 2018

alexcrichton deleted the wasm-custom-sections branch March 28, 2018 23:02

alexcrichton mentioned this pull request Apr 2, 2018

Stability of the wasm32-unknown-unknown target rustwasm/team#82

Closed

8 tasks

kennytm mentioned this pull request May 26, 2018

Tracking issue for the #[wasm_custom_section] attribute #51088

Closed

alexcrichton mentioned this pull request Jul 6, 2018

Tracking issue for #[wasm_import_module] #52090

Closed

pnkfelix mentioned this pull request Nov 4, 2019

Were all the run-make tests meant to to be moved to run-make-fulldeps? #66085

Closed

caspervonb mentioned this pull request Feb 1, 2020

Investigate ES6 modules as our internal components emscripten-core/emscripten#5828

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Add a `#[wasm_custom_section]` attribute #48883

rustc: Add a `#[wasm_custom_section]` attribute #48883

alexcrichton commented Mar 9, 2018

rust-highfive commented Mar 9, 2018

Mark-Simulacrum Mar 9, 2018

alexcrichton Mar 9, 2018

pepyakin Mar 9, 2018

pepyakin Mar 9, 2018

bors commented Mar 11, 2018

bors commented Mar 14, 2018

nikomatsakis commented Mar 14, 2018

nikomatsakis left a comment

nikomatsakis Mar 14, 2018

alexcrichton Mar 14, 2018

alexcrichton commented Mar 14, 2018

bors commented Mar 14, 2018

alexcrichton commented Mar 14, 2018

bors commented Mar 14, 2018

alexcrichton commented Mar 14, 2018

bors commented Mar 14, 2018

bors commented Mar 22, 2018

alexcrichton commented Mar 22, 2018

bors commented Mar 22, 2018

eddyb commented Mar 28, 2018

alexcrichton commented Mar 28, 2018

rustc: Add a #[wasm_custom_section] attribute #48883

rustc: Add a #[wasm_custom_section] attribute #48883

Conversation

alexcrichton commented Mar 9, 2018

rust-highfive commented Mar 9, 2018

Mark-Simulacrum Mar 9, 2018

Choose a reason for hiding this comment

alexcrichton Mar 9, 2018

Choose a reason for hiding this comment

pepyakin Mar 9, 2018

Choose a reason for hiding this comment

pepyakin Mar 9, 2018

Choose a reason for hiding this comment

bors commented Mar 11, 2018

bors commented Mar 14, 2018

nikomatsakis commented Mar 14, 2018

nikomatsakis left a comment

Choose a reason for hiding this comment

nikomatsakis Mar 14, 2018

Choose a reason for hiding this comment

alexcrichton Mar 14, 2018

Choose a reason for hiding this comment

alexcrichton commented Mar 14, 2018

bors commented Mar 14, 2018

alexcrichton commented Mar 14, 2018

bors commented Mar 14, 2018

alexcrichton commented Mar 14, 2018

bors commented Mar 14, 2018

bors commented Mar 22, 2018

alexcrichton commented Mar 22, 2018

bors commented Mar 22, 2018

eddyb commented Mar 28, 2018

alexcrichton commented Mar 28, 2018

rustc: Add a `#[wasm_custom_section]` attribute #48883

rustc: Add a `#[wasm_custom_section]` attribute #48883