Compilation to WASM?

[RReverser]

Miri maintainer note: this is a fun project, but not something we currently intend to support officially. To keep maintenance manageable, Miri only supports running on platforms that rustc supports running on.

Compiling the whole Rustc to WASM is a pretty big undertaking for many reasons.

However, Miri doesn't need an actual codegen and many other parts of the whole Rustc, so I wonder how realistic it would be to compile it and the pieces it depends on to WASM instead? Are there any obvious blockers?

Mostly opening this to gauge interest and estimate complexity, as I believe there is an interest in running Rust directly in the browser on playground-like websites.

P.S. Despite what I said in the first sentence, this was actually done for Clang a while ago - https://tbfleming.github.io/cib/ - which includes LLVM compiled to WASM that, in turn, generates more WASM dynamically during runtime. In theory, it should be possible to do the same for Rust, especially since they share LLVM, but for now having just an interpreter could already be an interesting starting goal.

[oli-obk]

I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata. That way we should be able to build a rustc which doesn't depend on llvm or other C code.

[bjorn3]

I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata.

I had created one in the past but it bitrotted and it was unused, so I removed it in rust-lang/rust#58847. If you copy https://github.com/bjorn3/rustc_codegen_cranelift/blob/11d816c/src/lib.rs#L182-L190 into provide and provide_extern it should work. (Also change target_features_whitelist to contain the same as cg_llvm https://github.com/rust-lang/rust/blob/2d401fb4dc89eaef5b8f31330636094f9c26b4c4/src/librustc_codegen_llvm/llvm_util.rs#L249, otherwise stdsimd wont compile.)

Another thing necessary is replacing the dlopen for loading the codegen backend with a regular extern crate. Eg replace the match at https://github.com/rust-lang/rust/blob/08bfe16129b0621bc90184f8704523d4929695ef/src/librustc_interface/util.rs#L271 with _ => || Box::new(MetadataOnlyCodegenBackend) or however you call the backend.

[bjorn3]

I want to do the same for https://github.com/bjorn3/rustc_codegen_cranelift/, but I want it to pass the rustc test suite first and cranelift doesnt support wasm output yet.

[RalfJung]

Intriguing. :) I should add one warning though: Miri isn't a fast interpreter. It's really slow. So I don't think it is actually a good environment to use to run code, I see it as more useful for debugging and testing.

But, don't let me stop you! I just felt I should give you a fair warning. And if ideas like this leak to people making Miri lightning fast while maintaining all the UB checking, I'll be even more happier. :D

[RReverser]

Miri isn't a fast interpreter. It's really slow. So I don't think it is actually a good environment to use to run code, I see it as more useful for debugging and testing.

That's understandable, but I think it's ought to be good enough for typical playground snippets :)

[RReverser]

I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata. That way we should be able to build a rustc which doesn't depend on llvm or other C code.

I guess that's one way, although I was wondering if Miri actually needs the main rustc crate or maybe it could be possible to depend only on some of the finer-grained rustc_* crates and avoid including codegen altogether?

[bjorn3]

The codegen backend is necessary for rustc_driver to work. Using a dummy codegen backend (<100LOC mostly copyable from the code I mentioned in #722 (comment)) is a lot easier than duplicating all the things rustc_driver does (>>1000LOC).

[RReverser]

Fair enough.

[bjorn3]

I am currently trying to compile rustc for wasm (https://github.com/bjorn3/rust/tree/compile_rustc_for_wasm), but I am hitting a compiler bug: rust-lang/rust#60540.

[RReverser]

@bjorn3 I've rebased your branch onto master, updated deps and fixed cfg's from target_env to target_os - you can check it out at https://github.com/RReverser/rust/tree/compile_rustc_for_wasm.

Eventually it compiled successfully, but then ran into the same runtime validation issue with invalid code generated by Rust.

However, I recompiled in release mode and then it passed validation!

That got me thinking it should work now, but running the generated file with wasmtime or wasmer now seems to just hang. Some infinite loop somewhere perhaps?

[RReverser]

@bjorn3 Oh... maybe it's just been taking so long (especially the compilation part). I've tried wasmer with --backend singlepass instead now, and it has actually worked!

$ ./wasmer run target/wasm32-unknown-wasi/release/rustc_binary.wasm --backend singlepass
Usage: rustc [OPTIONS] INPUT

Options:
    -h, --help          Display this message
        --cfg SPEC      Configure the compilation environment
    -L [KIND=]PATH      Add a directory to the library search path. The
                        optional KIND can be one of dependency, crate, native,
                        framework or all (the default).
    -l [KIND=]NAME      Link the generated crate(s) to the specified native
                        library NAME. The optional KIND can be one of
                        static, dylib, or framework. If omitted, dylib is
                        assumed.
        --crate-type [bin|lib|rlib|dylib|cdylib|staticlib|proc-macro]
                        Comma separated list of types of crates
                        for the compiler to emit
        --crate-name NAME
                        Specify the name of the crate being built
        --edition 2015|2018
                        Specify which edition of the compiler to use when
                        compiling code.
        --emit [asm|llvm-bc|llvm-ir|obj|metadata|link|dep-info|mir]
                        Comma separated list of types of output for the
                        compiler to emit
        --print [crate-name|file-names|sysroot|cfg|target-list|target-cpus|target-features|relocation-models|code-models|tls-models|target-spec-json|native-static-libs]
                        Comma separated list of compiler information to print
                        on stdout
    -g                  Equivalent to -C debuginfo=2
    -O                  Equivalent to -C opt-level=2
    -o FILENAME         Write output to <filename>
        --out-dir DIR   Write output to compiler-chosen filename in <dir>
        --explain OPT   Provide a detailed explanation of an error message
        --test          Build a test harness
        --target TARGET Target triple for which the code is compiled
    -W, --warn OPT      Set lint warnings
    -A, --allow OPT     Set lint allowed
    -D, --deny OPT      Set lint denied
    -F, --forbid OPT    Set lint forbidden
        --cap-lints LEVEL
                        Set the most restrictive lint level. More restrictive
                        lints are capped at this level
    -C, --codegen OPT[=VALUE]
                        Set a codegen option
    -V, --version       Print version info and exit
    -v, --verbose       Use verbose output

Additional help:
    -C help             Print codegen options
    -W help             Print 'lint' options and default settings
    -Z help             Print unstable compiler options
    --help -v           Print the full set of options rustc accepts

[bjorn3]

Oh... maybe it's just been taking so long (especially the compilation part).

Yes, it takes several minutes to compile it using wasmtime with cranelift as backend.

However, I recompiled in release mode and then it passed validation!

🎉 🎉 🎉

and it has actually worked!

I tried actually compiling something, but it errors with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: StringError("operation not supported on wasm yet") }', src/libcore/result.rs:999:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

I am currently trying to figure out were it errors.

[bjorn3]

Places needing patching:

• librustc_interface/util.rs: spawn_thread_pool must not spawn a thread.
• librustc/session/filesearch.rs: get_or_default_sysroot must not be called, as it needs std::env::current_exe. I am now passing an explicit --sysroot.
• librustc/session/mod.rs: build_session_ local working_dir is created using std::env::current_dir.

Edit: pushed bjorn3/rust@15f980f (based on @RReverser's branch).

Now it errors at

thread 'main' panicked at 'unknown codegen backend llvm', src/librustc_interface/util.rs:277:18

Which is expected, as I had to remove the codegen backend dynamic loader. Will try to get https://github.com/bjorn3/rustc_codegen_cranelift to work with it.

[RReverser]

Places needing patching

FWIW previously (before even filing this issue) I tried compiling rustc with Emscripten instead, which should, in theory, reduce number of these places to patch, as it supports a bit more than WASI does. Haven't gotten too far though, because I tried to build completely unpatched rustc and there were few things that still didn't compile and probably needed similar fixes as in your branch.

Which is expected, as I had to remove the codegen backend dynamic loader. Will try to get bjorn3/rustc_codegen_cranelift to work with it.

I thought the plan was to build it without any codegen, just with miri? Or do you want to build an actual full rustc?

[bjorn3]

I thought the plan was to build it without any codegen, just with miri? Or do you want to build an actual full rustc?

I want them both. :) I currently have rustc_codegen_cranelift hooked up, but rustc gives an error before rustc_codegen_cranelift can do actual codegen: can't find crate for `std` . Supporting miri will need that error to be fixed too.

[RReverser]

can't find crate for std

Yeah for that I think you'll need to do the proper build (via x.py build) to build all components. I haven't had much luck with that yet due to failures in other crates which probably also need to be patched similarly to rustc itself.

[bjorn3]

Yeah for that I think you'll need to do the proper build (via x.py build) to build all components.

Seems like it doesn't even reach the rustc version check for the libraries. I added --sysroot $(rustc --print sysroot) and disabled the rustc version check, but it still gives the same error.

[bjorn3]

Switching from wasmer to wasmtime fixed it. It even got to the beginning of codegen.

Edit: filled wasmerio/wasmer#434.

[bjorn3]

I am currently working on making miri compile for wasi, which this issue was actually about.

[bjorn3]

It seems to trap while calling the ecx.run. :(

error while processing main module ../../target/wasm32-unknown-wasi/release/rustc_binary.wasm: Instantiation error: Trap occurred while invoking start function: wasm trap at 0x2a881eb82

I pushed the wip stuff to my branch.

[RReverser]

@bjorn3 Left a comment on your MIRI commit on your branch.

[RReverser]

@bjorn3 But also, I'm not sure why rustc is now depending on miri... shouldn't it be the other way around? (like in non-WASI version)

[bjorn3]

I did that to be able to prevent having to recompile every rustc crate, which is slow and to prevent having to copy all files in the dir layout rustc wants a sysroot to be.

[RReverser]

I'm not sure I understand what you're saying... neither should be affected by which crate you compile as an entry point.

I've changed my local copy of Rust & MIRI to do just that, and got miri.wasm successfully, but yeah, also hitting some trap.

[bjorn3]

I meant that I had already compiled all crates in rust/target. When switching to miri as crate root, I would have to recompile all crates into miri/target. By keeping rustc-binary as crate root, I could reuse rust/target as target dir.

[beingminimal]

Compiling to working executables that can run in the browser doesn't work with this without a Webassembly backend for Cranelift (or a port of LLVM to run on wasm), but the equivalent of cargo check does work. For vscode.dev it would be necessary to write a vscode extension which uses vscode-wasm to run rustc.wasm. I have never written a vscode extension that uses vscode-wasm before so I don't know how hard that would be. I also don't know what the extension size limit is. The bundle of rustc.wasm + standard library is on the large side.

Hey thanks a lot for the reply.
We want to solve for people to write, compile and run rust code in vscode.dev which is the browser base.
I personally request you to study the links below. I think they have already done the required things to make this live.
We just need to connect available dots to achieve this.
https://code.visualstudio.com/blogs/2023/06/05/vscode-wasm-wasi
https://marketplace.visualstudio.com/items?itemName=ms-vscode.wasm-wasi-core
https://github.com/microsoft/vscode-wasm

I am not that much experienced otherwise i could have implemented a demo with these available things and have written blog also. But now it seems with your help it would be possible to achieve this.
A small step by step guide and github repo would be fine also.

Awaiting your reply. Thanks
@bjorn3 any thoughts bro?

[fee1-dead]

when I have time I might make a poc vscode-wasm extension

[jeff-hykin]

I have never written a vscode extension that uses vscode-wasm before so I don't know how hard that would be.

I have, its not hard cause its the same as the browser (unless VS Code has recently made changes to the API, which I doubt they would).

[bjorn3]

I did another build of rustc + cg_clif for wasm32-wasi with the latest rustc version and a couple of bugs fixed. The source code can be found at https://github.com/bjorn3/rust/tree/compile_rustc_for_wasm11 and the pre-compiled binaries at https://github.com/bjorn3/wasm-rustc (see comments.txt in the source tree for how to build it yourself) I have also updated the browser_wasi_shim example in bjorn3/browser_wasi_shim@9289c4c. It is now able to reach the exact point where the linker would be invoked again, rather than panicking during linking preparation. Compiling to wasm is tracked by bytecodealliance/wasmtime#2566.

[bjorn3]

Opened rust-lang/rust#120348 to upstream most of the build system changes, pushed rust-lang/rustc_codegen_cranelift@7d3b293 to upstream a cg_clif change and managed to somewhat reduce the diff on the compile_rustc_for_wasm13 branch. The second commit on that branch uses the wasm32-wasi-preview1-threads target to further reduce the diff, but requires wasi-threads, which browser_wasi_shim currently doesn't support.

[bjorn3]

rust-lang/rust#120348 has been merged. Opened rust-lang/rust#121392 to make patching away dlopen usage easier. I rebased the compile_rustc_for_wasm13 branch and removed a couple of unnecessary changes.

Edit: rust-lang/rust#121392 has been merged too. Rebased.

[bjorn3]

[jeff-hykin]

Got a link to the html/js for this?
This looks incredible even with all the caveats I'm sure it has!

[bjorn3]

I took https://github.com/bjorn3/browser_wasi_shim/blob/main/examples/rustc.html and made it invoke the miri I built from the second to last commit of the compile_rustc_for_wasm13 branch instead.

[jeff-hykin]

For others trying to run it, thats

• This commit: https://github.com/bjorn3/rust/tree/d8265e3fe341a6827ca97a062bcc4da6a25a4690
• And follow the instructions inside of comment.txt, but instead of the wasmtime step, take the generated dist folder and put it in the that webserver

[Nadrieril]

Somehow no one pasted this here: https://garriga.dev/rubri/

Rubri is a proof-of-concept wrapper for Miri that runs in the browser.

cc @LyonSyonII

[bjorn3]

New version available at the compile_rustc_for_wasm16 branch of https://github.com/bjorn3/rust. @oligamiq created a PR to use @whitequark's LLVM fork, so it should now be able to build for wasm, though I haven't tested linking yet. It also currently requires using wasm32-wasip1-threads, which doesn't work with browser_wasi_shim yet. You can still use the cg_clif backend with wasm32-wasip1 without threads instead by omitting --config config.llvm.toml when running ./x.py install.

[whitequark]

FWIW, I'm happy to see this effort and will support it from my side as much as I'm able to. Let me know if you need any further adjustments LLVM-side (though I'm sorta waiting on consensus on handling pthreads shims in wasi-libc at the moment).

[bjorn3]

The pthread shims thing (or adding wasip1-threads support to browser_wasi_shim) is the biggest blocker for getting this working in the browser. For complete end-to-end execution in the browser, lld would also need to be supported. I haven't checked if it already works though. Would need to rebuild LLVM again to check. And I did probably need to patch the rust standard library to support a custom host call for spawning processes such that rustc can actually invoke lld.

[oligamiq]

Regarding browser_wasi_shim's support for wasm32-wasip1-threads, it is currently under development. We're just one step away from the final stage.

rustc creates thread 1, and from there it generates 4 threads. However, after closing threads 2–5, it throws in thread 1, which causes the main wasm to not terminate. In order to terminate the currently running wasm along with its threads, the plan is to close all child threads when a throw occurs and make this observable. I plan to finish this tomorrow.

mpsc::channel is working properly. When outputting 1–1000 from both the main thread and the threads, the output alternates to some extent, so the functionality should be at a sufficient level.

However, since no thread pool has been created and files are accessible from multiple workers, the speed has significantly dropped...

I feel bad for Bjorn since I continued working after submitting the pull request when I made the file system accessible from multiple workers. Currently, I'm working on a different branch.
https://github.com/oligamiq/browser_wasi_shim/tree/wasi_multi_threads_rustc

※Require SharedArrayBuffer

[bjorn3]

I hadn't looked at your browser_wasi_shim PR yet as I assumed it to be under heavy development. Let me know when it is ready for review/ready for me to try it out.

[whitequark]

For complete end-to-end execution in the browser, lld would also need to be supported. I haven't checked if it already works though.

I've looked into that. Ideally, LLD would be built as a library and linked into the rustc binary. LLD already has a function you can call to run its main routine in-process, so you should probably use that instead of a hostcall; this has the advantage that the resulting rustc is runtime-agnostic. (I would be happy to ship yowasp-rustc for example! It would be great for firmware development.)

[bjorn3]

Ideally, LLD would be built as a library and linked into the rustc binary.

Do you have any pointers on how to do that?

I would be happy to ship yowasp-rustc for example! It would be great for firmware development.

Be aware that rustc.wasm is pretty large at 134MB. And it needs another 72MB for the standard library compiled to wasm32-wasip1-threads.

[whitequark]

Do you have any pointers on how to do that?

How familiar are you with LLVM? You're looking to call lld_main. If you can link to the right object, it seems self-explanatory after that.

Be aware that rustc.wasm is pretty large at 134MB. And it needs another 72MB for the standard library compiled to wasm32-wasip1-threads.

That sounds about right. I woulde expect LLD to add ~50M more, so ~250M before compression, maybe ~100M after. In my view this is still worth it since the object can be cached indefinitely at the client. The distribution infrastructure I'm using supports such large packages just fine.

Have you seen my FPGA toolchain integration with VS Code? The packages used in the demo are a ~50M download in total (from memory), which is a little high but is fine if you have broadband. ~100M of rustc isn't such a stretch, and I was planning on shipping yowasp-clang that would be of a similar size, anyway.

[bjorn3]

How familiar are you with LLVM? You're looking to call lld_main. If you can link to the right object, it seems self-explanatory after that.

Is enabling building of the lld standalone tool enough to build whatever static library I need to link against for lld_main to work?

[whitequark]

Is enabling building of the lld standalone tool enough to build whatever static library I need to link against for lld_main to work?

I think so, since the lld executable is a really thin wrapper (which is, iirc, autogenerated) over the function I linked to. But I would personally still scour the build system to see which target it is that does it.

[whitequark]

So after looking into it, I think you should probably build the lldWasm library and invoke lld::wasm::link, provided you can add arbitrary C++ (which IIRC you can in rustc). The lld_main I linked to earlier doesn't seem to be properly exported, and it also drags in ELF, COFF, and MachO linkers that you may not care about at all.

[bjorn3]

Managed to get linking using lld working: bjorn3/rust#8

[x0k]

Somehow no one pasted this here: https://garriga.dev/rubri/

I also use miri (and a few other solutions) here to run various programming languages in the browser: https://x0k.github.io/ppp/editor

[whitequark]

And you can use the YoWASP VS Code toolchain if you want a full editor in your browser when running programming language compilers.