Proposal to simplify create-obj / create-exe

[fschutt]

Right now the interaction between create-obj and create-exe is very cumbersome:

• create-obj outputs an object file and a static_defs.h file
• the object format (serialized or symbols) has to be tracked manually and re-entered when running create-exe
• the code paths are different for single-wasm vs multi-wasm binaries
• the code paths are also different for cross-compilation vs native compilation using cc

To solve these issues and reduce the amout of if / else in commands/create-exe.rs, I propose refactoring the code like this:

1. Always make create-obj output to a directory instead of to a file

wasmer create-exe $path

if path == folder: generate this layout in folder
        /extra_objects
            volume.obj (only present if input is a pirita file)
        /objects - compiled objects that are going to be linked together
            /atom1.o
            /atom2.o
        /headers (optional directory)
            /static_defs_atom1.h
            /static_defs_atom2.h
        /entrypoint.json - specifies commands with paths to the .o files
        wasmer_main.c -> wasmer_main.o -> wasmer_main.exe -> copied to --output
else if path == file
    -> create directory structure like above in a tempdir
    -> run wasmer create-exe again on that tempdir

2. Simplify wasmer create-obj:

   • Output object file will no longer be written to $OUTPUT_PATH, but to $OUTPUT_PATH/objects/{input_name}.o
   • --header-path will be removed (header will always be written to $OUTPUT_DIR/headers/static_defs_{input_name}.h)
   • this structure will be created even if we only compile a single .wasm file

3. Simplify object format serialized + object format symbols

   • Should not be specified as a CLI flag, but written to the entrypoint.json
   • Directory should "intelligently" know which object format the objects were compiled with

4. Simplify cross-compilation vs native compilation:

   • The only difference should be whether we use zig cc or the native CC (gcc, clang, etc) and zig build-exe or the native lld-link
   • Always create a cross-compilation target if --library-path, --zig-binary-dir or --target is set
   • If the target is the native host tripe: construct a link.tar.gz tarball from the WASMER_DIR, then compile using that --tarball

5. Simplify feature flags:

   • Demove static-artifact-create and webc_runner flags (always enabled)
   • Does this save us anything on compilation time / binary size?

[syrusakbary]

I think the proposal is more complicated than it should. Why? Because it forces us to save the whole directory structure on the backend. I'm aware we already do that, but it opens for other complications such as defining the entrypoint or doing other stuff.

Basically, create-exe should receive a wasm or webc (or wapm directory package that we can convert).

wasmer create-exe python.wasm # outputs always a .exe

As an optimization, we can call wasmer create-exe with the precompiled atoms

wasmer create-exe python.webc --precompiled-atom=sha1124234:python.o --precompiled-atom=sha23423:lib.o

The functions in each of the object files should probably be prefixed with the sha2342342 for each.

[syrusakbary]

Use case:

WABT

One package, multiple wasm files:

• wasm2wat.wasm
• wat2wasm.wasm
• wasm-strip.wasm

Steps

1. Create obj

The backend detects what are the atoms inside of the package: wat2wasm.wasm, wasm2wat.wasm, ... .
Then, it calls, for each atom:

wasmer create-obj wat2wasm.wasm --prefix=sha123123 -o /tmp/wat2wasm.o # the sha is SHA256(wat2wasm.wasm)

wasmer create-obj wasm2wat.wasm --prefix=sha223123 -o /tmp/wasm2wat.o # the sha is SHA256(wasm2wat.wasm)

wasmer create-obj myfile.webc --atom=wasm2wat --prefix=sha223123 -o /tmp/wasm2wat.o # the sha is SHA256(wasm2wat.wasm)

wasmer create-obj myfile.webc --prefix=sha223123 -o /tmp/wasm2wat.o # the sha is SHA256(wasm2wat.wasm)
# Error: specify the atom that you would like to compile

What is prefix?

Whenever the .o file is created with the prefix sha123123, it will have the functions:

• Wasm functions: wasmer_sha123123_function_1, wasmer_sha123123_function_2, ..
• Function Call trampolines: wasmer_sha123123_trampoline_function_call_1, wasmer_sha123123_trampoline_function_call_2, ...
• Dynamic Function trampolines: wasmer_sha123123_trampoline_dynamic_function_1, wasmer_sha123123_trampoline_dynamic_function_2, ...
• Metadata: wasmer_sha123123_METADATA

...

2. Generate C header file

wasmer generate-c-header wasm2wat.wasm --prefix=sha223123

2. Create exe

The backend detects what are the atoms inside of the package: wat2wasm.wasm, wasm2wat.wasm, ... .
The backend calls:

wasmer create-exe wabt.webc --precompiled-atom=ATOM_NAME:PREFIX:THE_PATH_OF_THE_OBJECT_FILE

Example:

wasmer create-exe wabt.webc --precompiled-atom=wat2wasm:sha123123:/tmp/wat2wasm.o --precompiled-atom=wasm2wat:sha223123:/tmp/wasm2wat.o

Only on the last step is where create-exe creates the internal directory (similar to what it's doing now):

• Static defs are created here