Seeking to Compile for Realtime Graphics on Mobile VR (Android) Devices, Please Advise

[KTRosenberg]

Summary

I am interested in using a high-performance wasm runtime on android. Specifically, I am targeting one of the mobile Oculus VR headsets, which uses an ARM CPU. I'd like to incorporate the C# version of wasmer into a Unity engine project. wasmer seems to target Linux, Mac, and Windows primarily, but I am curious whether what I am trying to do would be possible. May I have clarification? Can wasmer be used on android, or is it too reliant on system-specific libraries? How might I proceed?

Additional details

I am looking to compile code to wasm modules externally. I only need wasmer to run the wasm.
Also, for full transparency, is wasmer fast enough to meet the frame-to-frame requirements of a graphics application?

Thank you for your time.

[igrep]

I'm currently trying to run Wasmer on Android on AArch64.
I've modified Wasmer a little and confirmed it can run an extremely simple wasm file.
So I'm going to send a PR.
But currently Wasmer isn't available on Android. And I'm not sure it satisfies your requirement.

[MarkMcCaskey]

We are interested in supporting more platforms, I'm not familiar with the Android platform but it seems feasible -- I can do a bit of research on what it would take...

is wasmer fast enough to meet the frame-to-frame requirements of a graphics application?

Wasmer can be very fast for computation, the biggest issue you're likely to run into in the short term is if you want to compile a lot of code with optimizations dynamically (e.g. generating, compiling, and executing Wasm each frame) or if your application requires copying a large amount of memory between the host and guest -- that can be slow in some situations right now.

The first issue can be solved by doing the compilation asynchronously, the second will require some design to avoid if it's your bottleneck. I can respond with more specifics if you have an API in mind.

@igrep Awesome -- thanks! I look forward to seeing it

[KTRosenberg]

Yes, specifically I am hoping to run wasm in Unity on an Oculus Quest. I am not sure if good real-time performance is possible with so many “layers,” but it would be a major contribution to game developers to have this working.

I can try and follow-up with a rough/informal API sketch.

Regarding compilation, yes, that would need to be on a different thread.
As for memory copying, in javascript my understanding is that by importing memory created in js you can just read/write into the wasm memory block for free without copying. Is this impossible to do? To minimize back-and forth calls between the host (C#) and wasm, I would use a command buffer in wasm that stores an id corresponding to functions/interfaces defined in C#. The arguments/data would be copied into the wasm memory as a “payload.” At the end of the frame, I imagine the C# would just switch on the command buffer ids and run through the data.

This would amount to only one function call per frame.

I’m not sure if you need to do copying behind-the-scenes though.

[MarkMcCaskey]

Yeah, that's perfect! With Wasmer as well, the host can access Wasm memory directly, but accessing host memory from the Wasm module requires copying. I think it should work fine, when compiling with the LLVM backend in particular, we can get very good performance, if the Wasm module is compiled from something like C or Rust, then the speed of computation is near-native -- I don't have the precise numbers in front of me but I believe we've had benchmarks where it executed somewhere like 1.2-2x the speed of native C.

Our other compiler backends have faster compilation times but produce native code that executes more slowly. Optimization is something that is orthogonal to most use cases and can always be improved when it becomes high enough priority. So I don't think it's something you'll have to worry about here!

I agree with you, Wasm in games/game engines is super exciting and has a lot of potential to reduce vendor lock-in, make mod-creation easier and more flexible, and allow for new types of mods. I'd love to hear about any progress you make here!

[KTRosenberg]

It sounds like we could just pre-allocate all memory on the WASM side in a very large chunk so the game engine could just access that without any copies. Even if there is copying, I think it would be in the form of some sort of message passing to let the system know if some action needs to be taken.

I'm not really sure how to go about compiling this project to work on the android platform in Unity, but this could easily be tested, if I understand correctly, by just opening Unity and switching the build target to android. Actually, I could be the one to test android dlls if I have a little help setting up the C# version from someone who knows more about Wasmer. I'm definitely hoping this could work.

[MarkMcCaskey]

It sounds like we could just pre-allocate all memory on the WASM side in a very large chunk so the game engine could just access that without any copies. Even if there is copying, I think it would be in the form of some sort of message passing to let the system know if some action needs to be taken.

There's one issue with using Wasm memory: if the point is to run untrusted code, then you can't trust that the data isn't being updated maliciously. This means it's unsafe/unsound in general for a language like Rust to have references to higher level types in the Wasm memory at least while the Wasm is executing. If the Wasm calls into the host and the host validates the bytes it's casting into higher level types before using them, as long as the Wasm module doesn't execute, those bytes will remain under the full control of the host.

If you're dealing with things like integers which don't have invalid binary representations (that is, all bit patterns are valid) then the worst that can happen is that you get some incorrect data, but for more complex types and any "Object" (data with invariants), it's harder.

I'm a fan of the message passing style because it's simple. This is kind of unexplored territory so it remains to be seen what works best, but with message passing Wasm modules can maintain their own state of the world and the interface between host and guest is really small and easy to test and verify as safe.

I don't really have experience with Unity or Android; maybe a good first step would be to get Unity + Wasmer hello world working on Desktop and then get it working on Android. At least to me, that seems like that has fewer moving parts.

It looks like @igrep has a fork for running Wasmer on Android! We'd love to merge that into master, too

[KTRosenberg]

Hmm. I guess as a research minded person I am less worried about security. Ideally I could avoid copies and just write to memory directly without extra checks. The idea is that wasm code would be known to be trusted code by the game developer, and it would be on him/her to make sure it interfaces with the C# code as wanted. I (Both wasm and non-wasm sides would be under the programmer’s control.) Before worrying about security, I’d first like to see what all the options are.

Maybe I misunderstand, but I didn’t think wasm had access to the whole host/C# program state—just the wasm-allocated sandboxed memory. I imagine the programmer still needs to create explicit hooks that let the wasm access anything outside the wasm memory. Otherwise it seems safe. —or do you give the system more power than that?

Would it help if I come-up with a few (more specific) examples of what I’m hoping to see? I could do that sometime soon.

[MarkMcCaskey]

Oh, that's fine, you should just be explicit that it's for running trusted code! Trusted code simplifies things a lot. Yeah, the issue here is about using Wasm memory as host memory and having the Wasm module change the underlying bytes. If you have a way to implicitly or explicitly "lock" areas of memory, then it's not an issue at all. Alternatively, if the host never casts a pointer into wasm memory into a pointer of a more complex type, then this is also not an issue.

As an example, in Rust bool must have the bit pattern 1 or 0 with all leading bits as 0, anything else is undefined behavior. If you have a Wasm memory offset, 0x100, and you get (wasm_memory_ptr + 0x100) as *const bool and you deref it and it has the value 3, then you've just triggered undefined behavior.

Your case should work fine!

[KTRosenberg]

Ah yes, it’s trusted code.

I’m not too sure if I was completely clear about how I imagine to share memory between the host and wasm, so it might be good for me to come up with some more examples anyway. Regardless, it sounds like what I’m hoping to do is possible.

If I want to try installing in Unity on Windows for now, as suggested, is there anything I should know about installing the system? I assume the C# repo has more information about that.

[MarkMcCaskey]

I’m not too sure if I was completely clear about how I imagine to share memory between the host and wasm, so it might be good for me to come up with some more examples anyway. Regardless, it sounds like what I’m hoping to do is possible.

Yeah, it'll look pretty similar to any low level FFI API but with a few more restrictions. I

If I want to try installing in Unity on Windows for now, as suggested, is there anything I should know about installing the system? I assume the C# repo has more information about that.

I've actually never used Wasmer from dot net or C# so I have no idea, I don't think there should be any issues but feel free to let me know if there are.

[KTRosenberg]

Any luck with the android support? I don't see a specific branch for it, so maybe it's a fork.

[igrep]

Yes, the changes to run on Android is in my fork: https://github.com/wasmerio/wasmer/compare/master...igrep:android?expand=1

Currently what I've done is just adding target_os = "android" anywhere necessary to build.
So I'm still missing something (Sorry I'm not an expert of Android!), but easy examples should work!
Now I have several TODOs left before sending a PR:

• Pass tests.
• Use the latest version of libc after Add definitions in ucontext.h in Android for ARM, x86, and x86_64 rust-lang/libc#1622 is released.
• Use the latest version of wasmparser after Fix typo in ValidatingParser::current_position bytecodealliance/wasmparser#201 is released. (Or, is the misspelled method is overwritten by Improved Metering #1041?)
• Port some Linux-specific code to Android if necessary.

[MarkMcCaskey]

@igrep

Use the latest version of wasmparser after bytecodealliance/wasmparser#201 is released. (Or, is the misspelled method is overwritten by #1041?)

was fixed in #1261!

This is super exciting thanks for working on this!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Feel free to reopen the issue if it has been closed by mistake.