Take another step towards interface types #395
Conversation
This commit is another large refactoring on the tail of WebAssembly#391 which is intended to help march one step closer to using pure interface types for specifying WASI and using `*.witx` files. Contained here is a large amount of refactoring of the `*.witx` files, the internals of the `witx` crate, and how code generators are supposed to work. At the `*.witx` level, some notable changes have been made: * All functions returning results now return `(expected ok? (error $errno))`. This helps signify that the intention of all functions is that they return a packaged value which is either a successful result or the erroneous error code on failure. ABI-wise nothing has changed, but this is intended to model more closely what the interface-types API of this would be. * The `flags` type in `*.witx` now desugars as a struct-of-bools rather than simply an `int`. This doesn't have any effect on the ABI today, but it does change the AST and type-level representation of these types. All existing `*.witx` files in this repository have been updated for all phases with these new forms. To reiterate, though, no details of the ABI have changed (or at least not intentionally). Everything should still be ABI-compatible with before. Under the hood for code generators this is a very large breaking change. The intention of this commit is to start moving code generators relying on `witx` into the direction that we'll end up with interface types. Namely the `coretypes` module is entirely replaced with a new `abi` module with the intention of handling lots more high-level details of translation than the previous module did. The `InterfaceFunc` type now sports two new methods: `call_wasm` and `call_interface`. These two methods represent the two halves of lifting/lowering operations performed by code generators. The `call_wasm` function takes interface types values (or whatever they are represented as in the source language) and converts them to wasm types to call a wasm import. The wasm import's return values are then "deserialized" back into the language's interface types values too. The `call_interface` function is the opposite, it assumes that you're coming from wasm values and calling, e.g., a host function with interface values. The `abi` module is refactored with an `Instruction` `enum` which lists out what are the current set of "adapter instructions". These adapter instructions are intended to somewhat model the eventual idea of adapter functions and their instructions, but for now they're pretty specific to WASI as-is today. All instructions currently model what WASI currently does, sometimes in very specific manners. It's expected that the `Instruction` type will be in flux as we tweak the ABI of WASI over time, but the current set of instructions will likely only be expanded because of maintaining compatibility with the current snapshot. The expected usage of `Instruction` is that code generators will implement how to generate code for each `Instruction`. This should be a very low-level operation (generating code per instruction) and should be in theory quite easy to implement. Not all code generators need to implement all instructions depending on their use case. Additionally WASI as-is today doesn't always exercise all types of instructions. The next steps after a PR like this include starting to define a new ABI for WASI which supports all types in all positions rather than the current ABI which supports only a limited subset of types in some positions. The intention is that this new ABI may add a new instruction or two but will generally not be a large breaking change for all existing code generators. Some new additions are expected but other than that existing code generators updated to use this PR will require little effort to support new ABIs.
|
For a frame of reference, I've updated three different code generators for this refactoring of
My hope is that by updating a number of locations this stresses this implementation in a few ways. It first ensures that the actual ABI of all functions has not changed. Additionally it enables running tests and diffs to ensure that generated code has not radically changed shape and still does roughly the same thing. Finally it tests both sides of calling and defining WASI APIs. The PRs to all those repos do not exist just yet since the code is in a bit of a messy state. Additionally I'd like to perform another pass in this repository to document things before merging (it's just getting late here today). If desired though I can push up the work-in-progress state of each of the branches for reviewing if desired. |
|
This seems great! Will it be possible to have the C headers be completely unchanged by this? (That would give a fairly high degree of confidence that the ABI didn't break). |
|
Last time we made this sort of change (#220) we weren't able to keep the text of the headers the same but I used static assertions of size/align to show that the data layout was identical: WebAssembly/wasi-libc#165 |
|
Oh I'm not actually really that worried about this accidentally breaking the ABI we have today. I've changed how the header file is generated and the APIs it generates are indeed different, but that's intentional because shim functions are used to delegate to the real functions (and the real functions don't have a changed ABI). I mostly just wanted to emphasize that all the type-level and I've also finished up documenting remaining TODO items and adding some more tests I wanted to be sure to get to. @pchickey would you be ok merging this and publishing this in this state? If you'd prefer I'm also happy to clean up and publish the changes to the code generators to review first. It's worth mentioning that the code generators are seeing breaking changes in the code they generate, namely:
Note that in C's case the adapters are currently simple but it's expected that they'll get more complicated in the future. For example a next step would be to add debug-mode (or maybe always-on?) code which asserts that outgoing values are all valid. For example witx enums are represented in C as integers, but this means that invalid values could be passed to a consuming API. This error should ideally be caught in C rather than on the other side of the API to reflect the intended semantics of interface types. |
This pulls in the new `witx` 0.9 crate which notably includes WebAssembly/WASI#395. This is a large refactoring of how witx signatures are processed and a large update to the actual syntax of the witx ABIs themselves. This commit adjusts to all the new `witx` APIs and also tweaks idioms in a few locations as well. Code generation is now less custom to Rust and instead tries to match almost exactly what the `witx` crate tells us to generate. Additionally the representation of types is chosen to more closely align with interface types in the future where it's impossible to pass invalid values to the outside world. Notable changes here are: * `enum`-like values are now newtype wrappers around integers to prevent invalid values from flowing out. * Error traits are now implement directly for the `Errno` wrapper. * The module containing the raw functions now exclusively uses i32/i64 types and doesn't try to use convenience types in Rust.
This pulls in the new `witx` 0.9 crate which notably includes WebAssembly/WASI#395. This is a large refactoring of how witx signatures are processed and a large update to the actual syntax of the witx ABIs themselves. This commit adjusts to all the new `witx` APIs and also tweaks idioms in a few locations as well. Code generation is now less custom to Rust and instead tries to match almost exactly what the `witx` crate tells us to generate. Additionally the representation of types is chosen to more closely align with interface types in the future where it's impossible to pass invalid values to the outside world. Notable changes here are: * `enum`-like values are now newtype wrappers around integers to prevent invalid values from flowing out. * Error traits are now implement directly for the `Errno` wrapper. * The module containing the raw functions now exclusively uses i32/i64 types and doesn't try to use convenience types in Rust.
|
I'll take a look at the code generator changes today! |
This commit updates to the 0.9 version of the witx crate implemented in WebAssembly/WASI#395. This new version drastically changes code generation and how we interface with the crate. The intention is to abstract the code generation aspects and allow code generators to implement much more low-level instructions to enable more flexible APIs in the future. Additionally a bunch of `*.witx` files were updated in the WASI repository. It's worth pointing out, however, that `wasi-common` does not change as a result of this change. The shape of the APIs that we need to implement are effectively the same and the only difference is that the shim functions generated by wiggle are a bit different.
|
For posterity, these are the three other changes for this PR: |
Same as records, they use pointers
|
For completeness, POSIX generally requires |
|
FWIW another motivation for switching to trapping is that it reflects the intended final semantics of interface types themselves where adapter functions would be doing reads/writes when passing values between modules, and the adapter function would trap given an invalid pointer and such. Ok though I think this is in a good enough spot I'm going to merge with @pchickey's previous approval. |
|
Aha or not, I do not have the Green Button! @pchickey would you be ok merging/publishing for me? |
This commit updates to the 0.9 version of the witx crate implemented in WebAssembly/WASI#395. This new version drastically changes code generation and how we interface with the crate. The intention is to abstract the code generation aspects and allow code generators to implement much more low-level instructions to enable more flexible APIs in the future. Additionally a bunch of `*.witx` files were updated in the WASI repository. It's worth pointing out, however, that `wasi-common` does not change as a result of this change. The shape of the APIs that we need to implement are effectively the same and the only difference is that the shim functions generated by wiggle are a bit different.
|
published. @sunfishcode can you please add alex to committers here? |
This commit updates to the 0.9 version of the witx crate implemented in WebAssembly/WASI#395. This new version drastically changes code generation and how we interface with the crate. The intention is to abstract the code generation aspects and allow code generators to implement much more low-level instructions to enable more flexible APIs in the future. Additionally a bunch of `*.witx` files were updated in the WASI repository. It's worth pointing out, however, that `wasi-common` does not change as a result of this change. The shape of the APIs that we need to implement are effectively the same and the only difference is that the shim functions generated by wiggle are a bit different.
* Update witx to 0.9 This pulls in the new `witx` 0.9 crate which notably includes WebAssembly/WASI#395. This is a large refactoring of how witx signatures are processed and a large update to the actual syntax of the witx ABIs themselves. This commit adjusts to all the new `witx` APIs and also tweaks idioms in a few locations as well. Code generation is now less custom to Rust and instead tries to match almost exactly what the `witx` crate tells us to generate. Additionally the representation of types is chosen to more closely align with interface types in the future where it's impossible to pass invalid values to the outside world. Notable changes here are: * `enum`-like values are now newtype wrappers around integers to prevent invalid values from flowing out. * Error traits are now implement directly for the `Errno` wrapper. * The module containing the raw functions now exclusively uses i32/i64 types and doesn't try to use convenience types in Rust. * Update submodule management * Fix no-std-compat * Fix wasi-ephemeral * Regenerate lib_generated * Update to main branch * Update manifest directives
This commit updates to the 0.9 version of the witx crate implemented in WebAssembly/WASI#395. This new version drastically changes code generation and how we interface with the crate. The intention is to abstract the code generation aspects and allow code generators to implement much more low-level instructions to enable more flexible APIs in the future. Additionally a bunch of `*.witx` files were updated in the WASI repository. It's worth pointing out, however, that `wasi-common` does not change as a result of this change. The shape of the APIs that we need to implement are effectively the same and the only difference is that the shim functions generated by wiggle are a bit different.
This commit is another large refactoring on the tail of #391 which is
intended to help march one step closer to using pure interface types for
specifying WASI and using
*.witxfiles. Contained here is a largeamount of refactoring of the
*.witxfiles, the internals of thewitxcrate, and how code generators are supposed to work.
At the
*.witxlevel, some notable changes have been made:All functions returning results now return
(expected ok? (error $errno)). This helps signify that the intention of all functions isthat they return a packaged value which is either a successful result
or the erroneous error code on failure. ABI-wise nothing has changed,
but this is intended to model more closely what the interface-types
API of this would be.
The
flagstype in*.witxnow desugars as a struct-of-bools ratherthan simply an
int. This doesn't have any effect on the ABI today,but it does change the AST and type-level representation of these types.
All existing
*.witxfiles in this repository have been updated for allphases with these new forms. To reiterate, though, no details of the ABI
have changed (or at least not intentionally). Everything should still be
ABI-compatible with before.
Under the hood for code generators this is a very large breaking change.
The intention of this commit is to start moving code generators relying
on
witxinto the direction that we'll end up with interface types.Namely the
coretypesmodule is entirely replaced with a newabimodule with the intention of handling lots more high-level details of
translation than the previous module did.
The
InterfaceFunctype now sports two new methods:call_wasmandcall_interface. These two methods represent the two halves oflifting/lowering operations performed by code generators. The
call_wasmfunction takes interface types values (or whatever they arerepresented as in the source language) and converts them to wasm types
to call a wasm import. The wasm import's return values are then
"deserialized" back into the language's interface types values too. The
call_interfacefunction is the opposite, it assumes that you're comingfrom wasm values and calling, e.g., a host function with interface values.
The
abimodule is refactored with anInstructionenumwhich listsout what are the current set of "adapter instructions". These adapter
instructions are intended to somewhat model the eventual idea of adapter
functions and their instructions, but for now they're pretty specific to
WASI as-is today. All instructions currently model what WASI currently
does, sometimes in very specific manners. It's expected that the
Instructiontype will be in flux as we tweak the ABI of WASI overtime, but the current set of instructions will likely only be expanded
because of maintaining compatibility with the current snapshot.
The expected usage of
Instructionis that code generators willimplement how to generate code for each
Instruction. This should be avery low-level operation (generating code per instruction) and should be
in theory quite easy to implement. Not all code generators need to
implement all instructions depending on their use case. Additionally
WASI as-is today doesn't always exercise all types of instructions.
The next steps after a PR like this include starting to define a new ABI
for WASI which supports all types in all positions rather than the
current ABI which supports only a limited subset of types in some
positions. The intention is that this new ABI may add a new instruction
or two but will generally not be a large breaking change for all
existing code generators. Some new additions are expected but other than
that existing code generators updated to use this PR will require little
effort to support new ABIs.