Experimental Native Compilation Refactor #2272

syrusakbary · 2021-04-29T04:26:05Z

Description

This PR refactors the experimental native compilation, making it much more resilient.

It speeds up LLVM Native compilation by bypassing the single-threaded linking via LLVM, and emitting directly multiple object files (one per module). Therefore we speed it up via multi-threaded linking in via the lld linker (we can embed it in the future)
It refactors the output of the experimental native compilation to be just a struct with two fields:
1. A list of object file contents
2. The compilation frame info for each of the functions (containing the function length)
It parses the function size per module and outputs it in the frame info.

This PR is also designed in mind to make much easier the creation and parsing of SourceMaps that we can use later to detect traps in an exact location.

For the trap info to be completely binded we will need to merge this PR first:
#2250

And for traps to be working all the way on LLVM, we will need to merge this after:
#2216

Edit by @Hywan: Fixes #1727

Review

Add a short description of the change to the CHANGELOG.md file

# Conflicts: # lib/compiler-llvm/src/compiler.rs # lib/engine-native/src/artifact.rs

lib/compiler-llvm/src/compiler.rs

lib/engine-object-file/src/artifact.rs

lib/engine-native/src/serialize.rs

lib/compiler-llvm/src/compiler.rs

lib/compiler/src/compiler.rs

lib/compiler-llvm/src/compiler.rs

lib/compiler/src/compiler.rs

lib/engine-native/src/artifact.rs

lib/engine-native/src/serialize.rs

lib/engine-object-file/src/serialize.rs

Co-authored-by: nlewycky <[email protected]>

# Conflicts: # Cargo.lock # fuzz/Cargo.lock

# Conflicts: # Cargo.lock # fuzz/Cargo.lock # lib/engine-native/src/artifact.rs # lib/vm/src/trap/trapcode.rs

syrusakbary · 2021-04-30T20:13:52Z

lib/cli/src/commands/create_exe.rs

+            .args(link_aganist_extra_libs)
+            .arg("-o")
+            .arg(&self.output_path);
+        println!("COMMAND: {:?}", command);


TODO: Remove

lib/compiler-llvm/src/object_file.rs

nlewycky · 2021-05-03T19:45:57Z

lib/compiler/src/compiler.rs

+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExperimentalNativeCompilation {
+    /// The contents of emitted object files
+    pub object_files: Vec<Vec<u8>>,


Should the comment document the order of these? Is the order deterministic?

Yeah, the order is deterministic as the current implementation (per llvm-compilation), but IMHO should not be required since it's just an implementation detail?

Please comment that the order is irrelevant but deterministic. It helps the next person understand this data.

lib/engine-native/src/artifact.rs

lib/compiler-llvm/src/object_file.rs

lib/engine-object-file/src/artifact.rs

lib/engine-native/src/serialize.rs

nlewycky · 2021-05-03T20:18:52Z

lib/engine-native/src/serialize.rs

-    pub fn serialize(&mut self) -> Result<Vec<u8>, CompileError> {
+    /// Serialize the Metadata into bytes
+    /// The bytes will have the following format:
+    /// RKYV serialization (any length) + POS (8 bytes) + Endian (1 byte)


lib/engine-native/src/serialize.rs

lib/engine-object-file/src/serialize.rs

Co-authored-by: nlewycky <[email protected]>

# Conflicts: # Cargo.lock # fuzz/Cargo.lock # lib/compiler-llvm/Cargo.toml # lib/compiler-llvm/src/object_file.rs

nlewycky · 2021-05-04T23:24:02Z

lib/cli/src/commands/create_exe.rs

        #[cfg(not(windows))]
-        let wasm_object_path = PathBuf::from("wasm.o");
+        let wasm_object_path = PathBuf::from("wasm.a");
        #[cfg(windows)]
-        let wasm_object_path = PathBuf::from("wasm.obj");
+        let wasm_object_path = PathBuf::from("wasm.a");


No longer need windows/non-windows here since they're the same.

lib/engine-native/src/artifact.rs

lib/engine-native/src/serialize.rs

lib/engine-object-file/src/artifact.rs

lib/engine-object-file/src/serialize.rs

Co-authored-by: nlewycky <[email protected]>

# Conflicts: # Cargo.lock

Rkyv supports different endiands since v0.6.1 More info: rkyv/rkyv#104

…llvm-traps

syrusakbary · 2021-05-24T20:26:52Z

bors try

bors · 2021-05-24T20:35:11Z

try

Build failed:

Test on windows-x64

2342: Engine native (slim) r=syrusakbary a=syrusakbary  # Description This PR is the slim version of #2272. It advances on: * [x] Stops encoding the endian into the serialized data (rkyv already does that for us since `0.6.1`) * [x] Sets the proper function lengths in the native engine * [x] Catch signal traps only when the pc is originated from a Wasm function, and not in the host (fixing #2328)  # Review - [ ] Add a short description of the change to the CHANGELOG.md file Co-authored-by: Syrus <[email protected]> Co-authored-by: Syrus Akbary <[email protected]> Co-authored-by: Mark McCaskey <[email protected]>

syrusakbary added 5 commits April 26, 2021 09:05

WIP of LLVM metadata as a function generator

b2a41a7

Merge branch 'middleware-refactor' into llvm-traps

438ded1

# Conflicts: # lib/compiler-llvm/src/compiler.rs # lib/engine-native/src/artifact.rs

Merge branch 'master' into llvm-traps

cc74ba6

First working iteration of proper Native loading

fe05ed0

Refactored struct representation of experimental native compilation

6d1ca9e

syrusakbary requested review from Hywan, losfair, MarkMcCaskey and nlewycky as code owners April 29, 2021 04:26

Fixed linting issues

69f716d

syrusakbary commented Apr 29, 2021

View reviewed changes

lib/compiler-llvm/src/compiler.rs Show resolved Hide resolved

syrusakbary commented Apr 29, 2021

View reviewed changes

lib/engine-object-file/src/artifact.rs Outdated Show resolved Hide resolved

syrusakbary commented Apr 29, 2021

View reviewed changes

lib/engine-native/src/serialize.rs Outdated Show resolved Hide resolved

syrusakbary commented Apr 29, 2021

View reviewed changes

lib/engine-native/src/serialize.rs Show resolved Hide resolved

nlewycky reviewed Apr 29, 2021

View reviewed changes

Update lib/compiler/src/compiler.rs

5fa5379

Co-authored-by: nlewycky <[email protected]>

syrusakbary mentioned this pull request Apr 30, 2021

Scenario with very slow function execution in Wasmer on Windows compared to other runtimes and a native app #2260

Closed

syrusakbary and others added 7 commits April 29, 2021 18:23

Update lib/compiler-llvm/src/compiler.rs

5ca3d84

Co-authored-by: nlewycky <[email protected]>

Update wasmparser

7f726ca

Merge branch 'llvm-traps' of github.com:wasmerio/wasmer into llvm-traps

b97139a

Improved create-exe

f1d649d

Merge branch 'master' into llvm-traps

ca0da94

# Conflicts: # Cargo.lock # fuzz/Cargo.lock

Merge branch 'universal-compiled-info' into llvm-traps

91e24f0

# Conflicts: # Cargo.lock # fuzz/Cargo.lock # lib/engine-native/src/artifact.rs # lib/vm/src/trap/trapcode.rs

Updated serialization mechanism

e845d45

syrusakbary commented Apr 30, 2021

View reviewed changes

syrusakbary added 2 commits April 30, 2021 14:25

Attach frame info registration in native and object-file engine

b37981b

Merge branch 'master' into llvm-traps

727ec5a

nlewycky reviewed May 3, 2021

View reviewed changes

syrusakbary and others added 2 commits May 3, 2021 14:24

Update lib/engine-native/src/serialize.rs

26235a7

Co-authored-by: nlewycky <[email protected]>

Update lib/compiler-llvm/src/object_file.rs

37e566d

Co-authored-by: nlewycky <[email protected]>

syrusakbary and others added 7 commits May 3, 2021 14:29

Update lib/engine-object-file/src/artifact.rs

cc2cc77

Co-authored-by: nlewycky <[email protected]>

Improvoed comment

6eb7084

Reuse variable

555b16f

Fixed function body sizes

c352602

Update indexmap version

b183538

Merge branch 'master' into llvm-traps

e73ebb1

# Conflicts: # Cargo.lock # fuzz/Cargo.lock # lib/compiler-llvm/Cargo.toml # lib/compiler-llvm/src/object_file.rs

Fixed duplicated imports

8aa7247

nlewycky approved these changes May 5, 2021

View reviewed changes

syrusakbary and others added 3 commits May 4, 2021 19:00

Update lib/engine-object-file/src/artifact.rs

17b3b8e

Co-authored-by: nlewycky <[email protected]>

Update lib/engine-native/src/artifact.rs

fe3210a

Co-authored-by: nlewycky <[email protected]>

Update lib/engine-native/src/artifact.rs

355134e

Co-authored-by: nlewycky <[email protected]>

syrusakbary mentioned this pull request May 11, 2021

cranelift reports wrong trap types with dylib engine #1727

Closed

syrusakbary added 10 commits May 12, 2021 16:12

Merge branch 'master' into llvm-traps

1deb551

Added check for wasm pc (so we only trap on Wasm functions)

f86597e

Addressed all comments

6d02377

Make objectfile the same for windows and unix

d3a8d0b

Improve readability of frameinfo

d3c856b

Merge branch 'master' into llvm-traps

b900b5b

# Conflicts: # Cargo.lock

Fixed compilation issue with data

374da7b

Remove endianess check in engines

cd17339

Rkyv supports different endiands since v0.6.1 More info: rkyv/rkyv#104

Merge branch 'llvm-traps' of https://github.com/wasmerio/wasmer into …

360af93

…llvm-traps

Fixed linting

067391c

bors bot added a commit that referenced this pull request May 24, 2021

Try #2272:

c65722b

syrusakbary mentioned this pull request May 25, 2021

Engine native (slim) #2342

Merged

4 tasks

syrusakbary closed this Oct 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental Native Compilation Refactor #2272

Experimental Native Compilation Refactor #2272

syrusakbary commented Apr 29, 2021 •

edited by Hywan

Loading

syrusakbary Apr 30, 2021

nlewycky May 3, 2021

syrusakbary May 3, 2021

nlewycky May 4, 2021

nlewycky May 3, 2021

nlewycky May 4, 2021

syrusakbary commented May 24, 2021

bors bot commented May 24, 2021

Experimental Native Compilation Refactor #2272

Experimental Native Compilation Refactor #2272

Conversation

syrusakbary commented Apr 29, 2021 • edited by Hywan Loading

Description

Review

syrusakbary Apr 30, 2021

Choose a reason for hiding this comment

nlewycky May 3, 2021

Choose a reason for hiding this comment

syrusakbary May 3, 2021

Choose a reason for hiding this comment

nlewycky May 4, 2021

Choose a reason for hiding this comment

nlewycky May 3, 2021

Choose a reason for hiding this comment

nlewycky May 4, 2021

Choose a reason for hiding this comment

syrusakbary commented May 24, 2021

bors bot commented May 24, 2021

try

syrusakbary commented Apr 29, 2021 •

edited by Hywan

Loading