Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement support for multiple commands in one native executable #3430

Merged
merged 141 commits into from
Jan 5, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
bb2acdb
Add wabt for testing multi-command exe
fschutt Dec 16, 2022
9d9ebd4
Add failing unit test, create entrypoint.json and prepare linking
fschutt Dec 16, 2022
a7e537c
WIP: debug create-exe
fschutt Dec 16, 2022
77ff807
Rework changes to create-exe: only print to stdout when debug_dir set
fschutt Dec 19, 2022
d2b118f
Undo temp change to build-wasmer
fschutt Dec 19, 2022
1d4bc04
Fix make lint / remove unnecessary code
fschutt Dec 19, 2022
82bc742
Add initial implementation of Entrypoint::Multi
fschutt Dec 19, 2022
92daae5
Start moving conditionals of compile_zig() and link() into function i…
fschutt Dec 19, 2022
9795415
Refactor: split out functions into free function, simplify logic
fschutt Dec 20, 2022
03693d5
Simplify create-exe a lot
fschutt Dec 21, 2022
8ef5e23
Merge branch 'master' into multi-command-exe
fschutt Dec 21, 2022
bc7d125
Add compile_c back and add compilation for atoms
fschutt Dec 21, 2022
c92d8d3
Add compilation for single and multi-wasm inputs
fschutt Dec 21, 2022
b511943
Add CLI handling to wasmer_main.c
fschutt Dec 21, 2022
9d9a6d3
Re-add generate_wasmer_main_c_static
fschutt Dec 21, 2022
3672e03
Merge branch 'master' into multi-command-exe
fschutt Dec 22, 2022
c798683
Debug why linking WASMER_METADATA doesn't work for multiple files
fschutt Dec 22, 2022
75a7e68
Merge branch 'master' into multi-command-exe
fschutt Dec 22, 2022
b873f0f
Fix make lint
fschutt Dec 23, 2022
88f6933
Remove static_create_exe_main.c since it's duplicated
fschutt Dec 23, 2022
acdc7e2
Fix error when WASMER_DIR is not defined + prepare libwasmer-headless
fschutt Dec 23, 2022
358296f
Fix linking for --object-format serialized
fschutt Dec 23, 2022
8fbf051
Add fallback to system linker in case zig is not available
fschutt Dec 23, 2022
b06c04b
Ignore presence of zig if --use-system-linker is specified
fschutt Dec 23, 2022
62bdc08
Add prefix mapping to create-exe and create-obj
fschutt Dec 27, 2022
8e0484b
Add object output to create-obj
fschutt Dec 27, 2022
f6816dd
Add create_header_files_in_dir step
fschutt Dec 27, 2022
77bc458
Fix entrypoint serialization
fschutt Dec 27, 2022
3aa0ecb
Add option to reject multi-wasm files
fschutt Dec 27, 2022
f3e58cd
Fix errors in looking up prefix hashes
fschutt Dec 27, 2022
abac4ee
Fix most compilation errrors when migrating to prefix buildFix most c…
fschutt Dec 27, 2022
8a97f44
Debug why --object-format serialized segfaults
fschutt Dec 27, 2022
032cb2c
Fix segfault when using --object-format serialized and compile errors
fschutt Dec 28, 2022
798286b
Fix integration tests except for multi-command test
fschutt Dec 28, 2022
802e9e2
Fix create-exe bugs
fschutt Dec 29, 2022
994ea45
Merge branch 'master' into multi-command-exe
fschutt Dec 29, 2022
f7df7cc
Debug failing integration tests
fschutt Dec 29, 2022
e856932
Fix integration tests partially
fschutt Dec 29, 2022
ddc550b
Fix -target error
fschutt Dec 29, 2022
e00aae8
Fix the prefix / sha256 CLI parsing
fschutt Dec 29, 2022
f3d0902
Fix multi-command-exe test
fschutt Dec 29, 2022
49d2e14
Try running create-exe on Windows
fschutt Dec 29, 2022
cf81806
Fix wasmer_config_error integration test on Windows
fschutt Dec 29, 2022
ed251a4
Use strstr to detect WASI failure
fschutt Dec 29, 2022
8806c67
Fix actual error with strstr
fschutt Dec 29, 2022
1a572f9
Remove support for wasmer-headless for now, remove -lunwind
fschutt Dec 29, 2022
927feb3
Remove unnecessary format!() calls to fix errors on Windows
fschutt Dec 29, 2022
b5f1fc1
Get rid of format!() macros to debug wrong pathbufs on Windows
fschutt Dec 29, 2022
65bcdfb
Use splitn to resolve mapdir bug on Windows
fschutt Dec 29, 2022
9efd87c
Use format!() on all file paths to fix compile errors on Windows
fschutt Dec 29, 2022
aa3cacb
Normalize path
fschutt Dec 29, 2022
d8b5735
Fix std::fs::copy on Windows
fschutt Dec 29, 2022
76fb7e4
Normalize input path
fschutt Dec 29, 2022
fa8468c
Fix description for --precompiled-atom
fschutt Dec 30, 2022
bbf7870
Fix make lint
fschutt Dec 30, 2022
0ca9efd
Implement caching + add integration test
fschutt Dec 30, 2022
9df3d95
Fix make lint
fschutt Dec 30, 2022
cbc0775
Set object_format when creating module_infos
fschutt Dec 30, 2022
fd4cff1
Add debugging to zig compilation
fschutt Dec 30, 2022
6ceb013
Fix bug that accidentally deleted the output file after zig compile
fschutt Dec 30, 2022
cca97fb
Fix integration tests
fschutt Dec 30, 2022
07a42c4
Merge branch 'master' into multi-command-exe
fschutt Jan 2, 2023
8d0adfc
Fix integration tests
fschutt Jan 2, 2023
7f86f92
Fix error in Makefile
fschutt Jan 2, 2023
35ff802
Fix error in Makefile
fschutt Jan 2, 2023
1bc4cea
Enable integration tests on windows-x64
fschutt Jan 2, 2023
d88d95d
Fix CARGO_TARGET in Makefile
fschutt Jan 2, 2023
5ae8026
Fix merge error in make-package
fschutt Jan 2, 2023
b0141f9
Fix CARGO_TARGET_STR
fschutt Jan 2, 2023
cc6a74f
Fix CARGO_TARGET again
fschutt Jan 2, 2023
c1b2c96
Remove unnecessary clap args
fschutt Jan 2, 2023
5f87aa6
Add ENV to zig-binary-path
fschutt Jan 2, 2023
c450fc2
Remove duplicated code paths for create-exe
fschutt Jan 2, 2023
ddbe5d7
Fix typo conpile -> compile
fschutt Jan 2, 2023
de39d61
Use strip_prefix instead of replacen
fschutt Jan 2, 2023
26c4fbe
Use String::from_utf8_lossy
fschutt Jan 2, 2023
c8e7185
Proper parsing for zig version
fschutt Jan 2, 2023
015fc8c
Use extra constants for system libraries
fschutt Jan 2, 2023
b2d4e97
Remove unnecessary manual CLI parsing
fschutt Jan 2, 2023
1736a80
Fix make lint
fschutt Jan 2, 2023
4a0587f
Port fix for integration tests from fast-ci PR
fschutt Jan 2, 2023
1d7ffd9
Add integration tests to verify that create-obj works as expected
fschutt Jan 2, 2023
234f207
Fix filter_tarballs when downloading tarballs
fschutt Jan 2, 2023
a44928e
Try fixing cross-compile test again
fschutt Jan 2, 2023
b6d0800
Try fixing integration tests again
fschutt Jan 2, 2023
48769d6
Remove failing integration test
fschutt Jan 2, 2023
bab6be5
Remove test-setup test
fschutt Jan 2, 2023
c217b8d
Revert "Remove unnecessary manual CLI parsing"
fschutt Jan 2, 2023
f44c6bd
Try linking to -lunwind on Windows
fschutt Jan 2, 2023
a859d68
create-exe: use -lc on all platforms
fschutt Jan 2, 2023
73859f5
Use -OReleaseFast to disable exception handling on Windows
fschutt Jan 2, 2023
51b13f6
Remove -lunwind
fschutt Jan 2, 2023
4f2e14f
Merge branch 'master' into multi-command-exe
fschutt Jan 2, 2023
2c1eee4
Add -lunwind on non-windows targets
fschutt Jan 2, 2023
6e46e4f
Add CLI flag to select for a URL or version to download tarballs from
fschutt Jan 3, 2023
66f78ae
Add special-case compilation for windows and add comments
fschutt Jan 3, 2023
9cd8d09
Merge branch 'master' into multi-command-exe
fschutt Jan 3, 2023
4c61969
Add debugging for where the "path not found" error comes from
fschutt Jan 3, 2023
4d5c899
Add more debugging to debug "file not found" error
fschutt Jan 3, 2023
9310d9c
Force --test-threads=1 to debug continouusly
fschutt Jan 3, 2023
f34e12e
Fix panic if WASMER_DIR does not exist
fschutt Jan 3, 2023
d3f177b
Remove debugging println and capture stdout correctly
fschutt Jan 3, 2023
1dad984
Fix make lint
fschutt Jan 3, 2023
731a0e0
Disable tests on Windows
fschutt Jan 3, 2023
d166bf8
Windows: disable more tests due to -lunwind issue, fix include paths
fschutt Jan 3, 2023
23791ba
Fix wasmer init tests on Windows
fschutt Jan 3, 2023
cfe89e5
Disable more tests + fix make lint
fschutt Jan 3, 2023
e8989f3
Fix prefix issue in init-publish test on Windows
fschutt Jan 3, 2023
3a7307b
Add integration test to test that CLI args handling is done properly
fschutt Jan 4, 2023
afb9d40
Add gen-c-header option
fschutt Jan 4, 2023
5d95913
Add support for gen-c-header on pirita files
fschutt Jan 4, 2023
d8ca009
Add metadata length calculation to gen-c-header
fschutt Jan 4, 2023
683c60c
Refactor gen-c-header using Artifact::metadata
fschutt Jan 4, 2023
0fbcbee
Add debugging to why linux-x64 doesn't work
fschutt Jan 4, 2023
a344e94
Switch back to ReleaseSafe
fschutt Jan 4, 2023
ae08b67
Fix Github CI installing choco / LLVM
fschutt Jan 4, 2023
1cb7b06
Add LLVM_DIR/bin to GitHub path
fschutt Jan 4, 2023
219151f
Try using prefixer = None like in master
fschutt Jan 4, 2023
743660c
Debug why llvm-config isn't found on Windows
fschutt Jan 4, 2023
73c4017
Set ENABLE_LLVM=true on Windows
fschutt Jan 4, 2023
5e096a0
CARGO_TARGET_STR -> CARGO_TARGET_FLAG
fschutt Jan 4, 2023
aefe5fc
Merge branch 'master' into multi-command-exe
fschutt Jan 4, 2023
253cd21
Document metadata_prefix and remove PrefixerFn
fschutt Jan 4, 2023
d5062c1
Add link to GitHub issue on why certain Windows test are ignored
fschutt Jan 4, 2023
2c9ad00
WASMER_{}_METADATA -> WASMER_METADATA_{}
fschutt Jan 4, 2023
398c138
Set LLVM_SYS_120_PREFIX on build-wasmer step
fschutt Jan 4, 2023
5df8789
Add "Metadata" symbol to Symbol enum
fschutt Jan 4, 2023
b963d2c
Remove prefix from Symbol::Metadata
fschutt Jan 4, 2023
84dc64b
Migrate from WASMER_METADATA_ to using SymbolRegistry
fschutt Jan 4, 2023
30fed6f
Remove enabling LLVM by force
fschutt Jan 4, 2023
697936f
Install LLVM on Windows via URL instead of choco
fschutt Jan 4, 2023
bf37e63
Remove last case of manual WASMER_METADATA formatting and enable LLVM
fschutt Jan 4, 2023
958a741
Fix LLVM WASMER_METADATA bug
fschutt Jan 4, 2023
3340574
Fix duplicated WASMER_METADATA logic
fschutt Jan 4, 2023
e32159e
Add libtinfo5 to Linux CI
fschutt Jan 4, 2023
15c1efd
Fix error when zig is not present, don't pass -target to linker
fschutt Jan 5, 2023
5a9418a
Disable publish_eh_frame
fschutt Jan 5, 2023
0313fe4
Revert "Disable publish_eh_frame"
fschutt Jan 5, 2023
ebcd97c
Fixed EH_Frame generation for LLVM Object files
ptitSeb Jan 5, 2023
b444386
Fixed a typo
ptitSeb Jan 5, 2023
d516713
Fixed another typo
ptitSeb Jan 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions lib/cli/src/c_gen/staticlib_header.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,11 @@ fn gen_helper_functions(atom_name: &str, module_name: &str) -> String {

/// Generate the header file that goes with the generated object file.
pub fn generate_header_file(
metadata_prefix: &str,
atom_name: &str,
module_info: &ModuleInfo,
symbol_registry: &dyn SymbolRegistry,
metadata_length: usize,
) -> String {
let module_name = format!("WASMER_METADATA_{}", metadata_prefix.to_uppercase());
let atom_name = metadata_prefix;
let mut c_statements = vec![
CStatement::LiteralConstant {
value: "#include \"wasmer.h\"\n#include <stdlib.h>\n#include <string.h>\n\n"
Expand All @@ -97,7 +95,7 @@ pub fn generate_header_file(
})),
},
CStatement::Declaration {
name: module_name.to_string(),
name: symbol_registry.symbol_to_name(Symbol::Metadata),
is_extern: true,
is_const: true,
ctype: CType::Array {
Expand Down Expand Up @@ -292,7 +290,7 @@ pub fn generate_header_file(
}

c_statements.push(CStatement::LiteralConstant {
value: gen_helper_functions(atom_name, &module_name),
value: gen_helper_functions(atom_name, &symbol_registry.symbol_to_name(Symbol::Metadata)),
});

c_statements.push(CStatement::LiteralConstant {
Expand Down
15 changes: 11 additions & 4 deletions lib/cli/src/commands/create_exe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ use std::process::Command;
use std::process::Stdio;
use wasmer::*;
use wasmer_object::{emit_serialized, get_object_for_target};
use wasmer_types::ModuleInfo;
use wasmer_types::compilation::symbols::ModuleMetadataSymbolRegistry;
use wasmer_types::{ModuleInfo, SymbolRegistry};
use webc::{ParseOptions, WebCMmap};

const LINK_SYSTEM_LIBRARIES_WINDOWS: &[&str] = &["userenv", "Ws2_32", "advapi32", "bcrypt"];
Expand Down Expand Up @@ -767,7 +768,10 @@ fn compile_atoms(
writer.flush()?;
}
ObjectFormat::Serialized => {
let module_name = format!("WASMER_METADATA_{}", prefix.to_uppercase());
let module_name = ModuleMetadataSymbolRegistry {
prefix: prefix.clone(),
}
.symbol_to_name(wasmer_types::Symbol::Metadata);
let module = Module::from_binary(&store, data).context("failed to compile Wasm")?;
let bytes = module.serialize()?;
let mut obj = get_object_for_target(target.triple())?;
Expand Down Expand Up @@ -991,7 +995,6 @@ pub(crate) fn create_header_files_in_dir(
prefixes: &[String],
) -> anyhow::Result<()> {
use object::{Object, ObjectSection};
use wasmer_types::compilation::symbols::ModuleMetadataSymbolRegistry;

if entrypoint.object_format == ObjectFormat::Serialized {
write_entrypoint(directory, entrypoint)?;
Expand Down Expand Up @@ -1404,7 +1407,11 @@ fn generate_wasmer_main_c(
)
})?;
let atom_name = prefix.clone();
let module_name = format!("WASMER_METADATA_{}", prefix.to_uppercase());

let module_name = ModuleMetadataSymbolRegistry {
prefix: prefix.clone(),
}
.symbol_to_name(wasmer_types::Symbol::Metadata);

if compile_static {
extra_headers.push(format!("#include \"static_defs_{atom_name}.h\""));
Expand Down
2 changes: 1 addition & 1 deletion lib/compiler/src/engine/artifact.rs
Original file line number Diff line number Diff line change
Expand Up @@ -571,7 +571,7 @@ impl Artifact {
let mut obj = get_object_for_target(target_triple).map_err(to_compile_error)?;

let object_name = format!(
"WASMER_METADATA_{}",
"WASMER_{}_METADATA",
fschutt marked this conversation as resolved.
Show resolved Hide resolved
fschutt marked this conversation as resolved.
Show resolved Hide resolved
metadata_prefix.unwrap_or_default().to_uppercase()
);
emit_data(&mut obj, object_name.as_bytes(), &metadata_binary, 1)
Expand Down