Document guidance around multithreading and Wasmtime

This commit writes a page of documentation for the Wasmtime book to
serve as guidance for embedders looking to add multithreading with
Wasmtime support. As always with any safe Rust API this reading is
optional because you can't mis-use Wasmtime without `unsafe`, but I'm
hoping that this documentation can serve as a point of reference for
folks who want to add multithreading but are confused/annoyed that
Wasmtime's types do not implement the `Send` and `Sync` traits.

Closes bytecodealliance#793

docs/SUMMARY.md

@@ -17,6 +17,7 @@
     - [Linking Modules](./examples-rust-linking.md)
     - [Debugging](./examples-rust-debugging.md)
     - [Using Multi-Value](./examples-rust-multi-value.md)
+    - [Multi-threading](./examples-rust-multithreading.md)
   - [Embedding in C](./examples-c-embed.md)
     - [Hello, World!](./examples-c-hello-world.md)
     - [Calculating the GCD](./examples-c-gcd.md)

docs/examples-rust-multithreading.md

@@ -0,0 +1,147 @@
+# Multi-threading
+
+When using Rust you're effectively immune from a whole class of threading issues
+such as data races due to the inherent checks in the compiler and traits like
+`Send` and `Sync`. The `wasmtime` API, like other safe Rust APIs, is 100% safe
+to use relative to threading if you never have any `unsafe` yourself. In
+addition to all of this, however, it's important to be aware of the limitations
+of `wasmtime` types and how this might affect your embedding use case.
+
+## Types that are `Send` and `Sync`
+
+Wasmtime has a number of types which implement both the `Send` and `Sync`
+traits:
+
+* [`Config`](https://docs.wasmtime.dev/api/wasmtime/struct.Config.html)
+* [`Engine`](https://docs.wasmtime.dev/api/wasmtime/struct.Engine.html)
+* [`Module`](https://docs.wasmtime.dev/api/wasmtime/struct.Module.html)
+* [`Trap`](https://docs.wasmtime.dev/api/wasmtime/struct.Trap.html)
+* [`InterruptHandle`](https://docs.wasmtime.dev/api/wasmtime/struct.InterruptHandle.html)
+* Type-descriptions of items
+  * [`ValType`](https://docs.wasmtime.dev/api/wasmtime/struct.ValType.html)
+  * [`ExportType`](https://docs.wasmtime.dev/api/wasmtime/struct.ExportType.html)
+  * [`ExternType`](https://docs.wasmtime.dev/api/wasmtime/struct.ExternType.html)
+  * [`ImportType`](https://docs.wasmtime.dev/api/wasmtime/struct.ImportType.html)
+  * [`FuncType`](https://docs.wasmtime.dev/api/wasmtime/struct.FuncType.html)
+  * [`GlobalType`](https://docs.wasmtime.dev/api/wasmtime/struct.GlobalType.html)
+  * [`MemoryType`](https://docs.wasmtime.dev/api/wasmtime/struct.MemoryType.html)
+  * [`ModuleType`](https://docs.wasmtime.dev/api/wasmtime/struct.ModuleType.html)
+  * [`TableType`](https://docs.wasmtime.dev/api/wasmtime/struct.TableType.html)
+  * [`InstanceType`](https://docs.wasmtime.dev/api/wasmtime/struct.InstanceType.html)
+
+These types, as the traits imply, safe to send and share across threads. Note
+that the major types to call out here are `Module` and `Engine`. The `Engine` is
+important because it enables sharing compilation configuration for an entire
+application. Each `Engine` is intended to be long-lived for this reason.
+
+Additionally `Module`, the compiled version of a WebAssembly module, is safe to
+send and share across threads. This notably means that you can compile a module
+once and then instantiate it on multiple threads simultaneously. There's no need
+to recompile a module on each thread.
+
+## Types that are neither `Send` nor `Sync`
+
+Wasmtime also has a number of types which are thread-"unsafe". These types do
+not have the `Send` or `Sync` traits implemented which means that you won't be
+able to send them across threads by default.
+
+* [`Store`](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html)
+* [`Linker`](https://docs.wasmtime.dev/api/wasmtime/struct.Linker.html)
+* [`Instance`](https://docs.wasmtime.dev/api/wasmtime/struct.Instance.html)
+* [`Extern`](https://docs.wasmtime.dev/api/wasmtime/struct.Extern.html)
+* [`Func`](https://docs.wasmtime.dev/api/wasmtime/struct.Func.html)
+* [`Global`](https://docs.wasmtime.dev/api/wasmtime/struct.Global.html)
+* [`Table`](https://docs.wasmtime.dev/api/wasmtime/struct.Table.html)
+* [`Memory`](https://docs.wasmtime.dev/api/wasmtime/struct.Memory.html)
+* [`Val`](https://docs.wasmtime.dev/api/wasmtime/struct.Val.html)
+* [`ExternRef`](https://docs.wasmtime.dev/api/wasmtime/struct.ExternRef.html)
+
+These types are all considered as "connected to a store", and everything
+connected to a store is neither `Send` nor `Sync`. The Rust compiler will not
+allow you to have values of these types cross thread boundaries or get shared
+between multiple threads. Doing so would require some form of `unsafe` glue.
+
+It's important to note that the WebAssembly specification itself fundamentally
+limits some of the concurrent possibilities here. For example it's not allowed
+to concurrently call `global.set` or `table.set` on the same global/table. This
+means that Wasmtime is designed to prevent at the very least concurrent usage of
+these primitives.
+
+Apart from the WebAssembly specification, though, Wasmtime additionally has some
+fundamental design decision which results in these types not implementing either
+`Send` or `Sync`:
+
+* All objects are independently-owned `'static` values that internally retain
+  anything necessary to implement the API provided. This necessitates some form
+  of reference counting, and also requires the usage of non-atomic reference
+  counting. Once reference counting is used Rust only allows shared references
+  (`&T`) to the internals, and due to the wasm restriction of disallowing
+  concurrent usage non-atomic reference counting is used.
+
+* Insertion of user-defined objects into `Store` does not require all objects to
+  be either `Send` or `Sync`. For example `Func::wrap` will insert the
+  host-defined function into the `Store`, but there are no extra trait bounds on
+  this. Similar restrictions apply to `Store::set` as well.
+
+* The implementation of `ExternRef` allows arbitrary `'static` types `T` to get
+  wrapped up and is also implemented with non-atomic reference counting.
+
+Overall the design decisions of Wasmtime itself leads all of these types to not
+implement either the `Send` or `Sync` traits.
+
+## Multithreading without `Send`
+
+Due to the lack of `Send` on types like `Store` and everything connected, it's
+not always as trivial to add multithreaded execution of WebAssembly to an
+embedding of Wasmtime as it is for other Rust code in general. The exact way
+that multithreading could work for you depends on your specific embedding, but
+some possibilities include:
+
+* If your workload involves instantiating a singular wasm module on a separate
+  thread, then it will need to live on that thread and communicate to other
+  threads via threadsafe means (e.g. channels, locks/queues, etc).
+
+* If you have something like a multithreaded web server, for example, then the
+  WebAssembly executed for each request will need to live within the thread that
+  the original `Store` was created on. This could be multithreaded, though, by
+  having a pool of threads executing WebAssembly. Each request would have a
+  scheduling decision of which pool to route to which would be up to the
+  application. In situations such as this it's recommended to [enable fuel
+  consumption](https://docs.wasmtime.dev/api/wasmtime/struct.Config.html#method.consume_fuel)
+  as well as [yielding when out of
+  fuel](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.out_of_fuel_async_yield).
+  This will ensure that no one request entirely hogs a thread executiong
+  WebAssembly and all requests scheduled onto that thread are able to execute.
+  It's also worth pointing out that the threads executing WebAssembly may or may
+  not be the same as the threads performing I/O for your server requests.
+
+* If absolutely required Wasmtime is engineered such that it is dynamically safe
+  to move a `Store` as a whole to a separate thread. This option is not
+  recommended due to its complexity, but it is one that Wasmtime tests in CI and
+  considers supported. The principle here is that all objects connected to a
+  `Store` are safe to move to a separate thread *if and only if*:
+
+  * All objects are moved all at once. For example you can't leave behind
+    references to a `Func` or perhaps a `Store` in TLS.
+
+  * All host objects living inside of a store (e.g. those inserted via
+    `Store::set` or `Func::wrap`) implement the `Send` trait.
+
+  If these requirements are met it is technically safe to move a store and its
+  objects between threads. The reason that this strategy isn't recommended,
+  however, is that you will receive no assistance from the Rust compiler in
+  verifying that the transfer across threads is indeed actually safe. This will
+  require auditing your embedding of Wasmtime itself to ensure it meets these
+  requirements.
+
+  It's important to note that the requirements here also apply to the futures
+  returned from `Func::call_async`. These futures are not `Send` due to them
+  closing over `Store`-related values. In addition to the above requirements
+  though to safely send across threads embedders must *also* ensure that any
+  host futures returned from `Func::wrapN_async` are actually `Send` and safe to
+  send across threads. Again, though, there is no compiler assistance in doing
+  this.
+
+Overall the recommended story for multithreading with Wasmtime is "don't move a
+`Store` between threads" and to architect your application around this
+assumption.