New Fuzz Target for Exercising Interface Functions #4307
Comments
|
We might actually want to have two instances of the Wasm component, so that values flow like so that we are exercising component-to-component trampolines and all that. |
|
Thanks, for writing this up, @fitzgen. A few questions for you:
|
Yeah, I think it makes sense to unblock this before working on it. We could probably technically implement some (most?) of the test case generator without the oracles, but I think it would be best if they were developed in together since I think the shape of the generator may be informed by the oracles.
I think a |
|
@alexcrichton Are you planning to create an issue for Also, it sounds like we might also need an issue for the interface-type-equivalent of |
|
I do indeed want to open an issue! Sorry I got caught up all day in a debugging adventure but that's over now so I'll go make an issue next. |
alexcrichton
added
the
wasm-proposal:component-model
|
FYI, I'm planning to start working on this tomorrow, assuming no significant additional work is needed on #4442. |
|
Looks like #4442 needs more work, so I won't be able to tackle this one quite yet. |
|
Quick question: Should the test case generator deliberately generate instances and components which do not match the generated type? For example, if the generated type is a record with three fields, should the instance generator (sometimes) generate values with too many, not enough, or mis-typed field values (or values which are instances of an entirely different sort of type), and should the component generator (sometimes) generate a component which expects a different record type (or a different sort of type)? |
|
You could do that with very low probability (something like |
|
Regarding the static API oracle: neither |
|
Yeah that sounds right to me. We'll want the seed of the byte slice to be deterministic somewhat in the sense that if something hits a bug on oss-fuzz we want to be able to reproduce it locally by overriding the seed. For now I think having an optional env var to specify the seed would be fine, we can always fiddle with it later too. |
|
I'm making good progress on this, but getting hung up on the asymmetry between the My first thought was to create my own type wrapper (e.g. @alexcrichton I think life would be a lot easier if |
|
That actually seems quite reasonable to me, the |
I tried to do this, but given that |
|
Regarding the above, would it be reasonable to add a |
|
I think |
Oh, good call. I missed that |
This commit implements the most general case of variants for fused adapter trampolines. Additionally a number of other primitive types are filled out here to assist with testing variants. The implementation internally was relatively straightforward given the shape of variants, but there's room for future optimization as necessary especially around converting locals to various types. This commit also introduces a "one off" fuzzer for adapters to ensure that the generated adapter is valid. I hope to extend this fuzz generator as more types are implemented to assist in various corner cases that might arise. For now the fuzzer simply tests that the output wasm module is valid, not that it actually executes correctly. I hope to integrate with a fuzzer along the lines of bytecodealliance#4307 one day to test the run-time-correctness of the generated adapters as well, at which point this fuzzer would become obsolete. Finally this commit also fixes an issue with `u8` translation where upper bits weren't zero'd out and were passed raw across modules. Instead smaller-than-32 types now all mask out their upper bits and do sign-extension as appropriate for unsigned/signed variants.
This is the first part of my work to address bytecodealliance#4307. We now generate 1000 arbitrary types and tests for those types at build time. Each test includes a component which imports and exports functions that take and return its respective type. The exported function calls the imported function, which is implemented by the host, and the host verifies that both the host function argument and the guest function return value match the original input value. In terms of bytecodealliance#4307, this includes the test case generator and the static API oracle. I'll follow up with a dynamic API oracle in a subsequent PR. Signed-off-by: Joel Dice <[email protected]>
* Implement variant translation in fused adapters This commit implements the most general case of variants for fused adapter trampolines. Additionally a number of other primitive types are filled out here to assist with testing variants. The implementation internally was relatively straightforward given the shape of variants, but there's room for future optimization as necessary especially around converting locals to various types. This commit also introduces a "one off" fuzzer for adapters to ensure that the generated adapter is valid. I hope to extend this fuzz generator as more types are implemented to assist in various corner cases that might arise. For now the fuzzer simply tests that the output wasm module is valid, not that it actually executes correctly. I hope to integrate with a fuzzer along the lines of #4307 one day to test the run-time-correctness of the generated adapters as well, at which point this fuzzer would become obsolete. Finally this commit also fixes an issue with `u8` translation where upper bits weren't zero'd out and were passed raw across modules. Instead smaller-than-32 types now all mask out their upper bits and do sign-extension as appropriate for unsigned/signed variants. * Fuzz memory64 in the new trampoline fuzzer Currently memory64 isn't supported elsewhere in the component model implementation of Wasmtime but the trampoline compiler seems as good a place as any to ensure that it at least works in isolation. This plumbs through fuzz input into a `memory64` boolean which gets fed into compilation. Some miscellaneous bugs were fixed as a result to ensure that memory64 trampolines all validate correctly. * Tweak manifest for doc build
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]> fix build error and type name tracking This fixes two issues: - build.rs was failing to build when the component_model feature was not enabled - Name assignments for types in component_types.rs were not being tracked properly because structurally identical types were being considered equal to each other. Signed-off-by: Joel Dice <[email protected]>
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]> fix build error and type name tracking This fixes two issues: - build.rs was failing to build when the component_model feature was not enabled - Name assignments for types in component_types.rs were not being tracked properly because structurally identical types were being considered equal to each other. Signed-off-by: Joel Dice <[email protected]>
|
@alexcrichton Things are coming together nicely. The fuzz tests have found a few bugs, which I've been debugging and fixing. I could use your help with dependency management, though. The publish CI job isn't happy with the "component-test-util/component-model" line I added to Cargo.toml, and my cargo skills aren't sharp enough to know how to address it. In this case |
|
Nice! I'm also happy to help out with debugging if you'd like as well. Otherwise though I think the best fix is to just unconditionally enable the I'll try to work on a PR to unconditionally enable the |
|
Here's a fun one: |
|
Ah yeah for that we'll want to constrain the I suppose another idea is to use the binary encoding instead of the text encoding of components since that specific error is coming from the ... This makes me think that we may stack overflow with super deep type hierarchies now... |
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]>
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]>
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]>
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]>
This addresses bytecodealliance#4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]>
This addresses #4307. For the static API we generate 100 arbitrary test cases at build time, each of which includes 0-5 parameter types, a result type, and a WAT fragment containing an imported function and an exported function. The exported function calls the imported function, which is implemented by the host. At runtime, the fuzz test selects a test case at random and feeds it zero or more sets of arbitrary parameters and results, checking that values which flow host-to-guest and guest-to-host make the transition unchanged. The fuzz test for the dynamic API follows a similar pattern, the only difference being that test cases are generated at runtime. Signed-off-by: Joel Dice <[email protected]>
|
This is done as of #4537, I believe. |
|
Indeed, and thanks again @dicej! |
To help make the component model and interface types production ready, we want
to extensively fuzz and exercise Wasmtime's ability to pass interface values
back and forth with Wasm.
This fuzz target should:
T,T -> T, andT -> Tthat passes its argument to the import functionand returns the result of the import function,
T,original value,
equal to the original value,
This tests that we can round trip interface values of type
Tboth as argumentsand returns to and from Wasm.
Here is a diagram showing what the fuzz target should do:
Test Case Generator
The test case generator will be responsible for generating
T,T, passes them to the importedhost function, and returns the result of the host function,
T.Oracle
There are two versions of the oracle:
and
Dynamic API Oracle
The dynamic API oracle will be paired with the test case generator at runtime
and use Wasmtime's dynamic API for interface type values (doesn't exist at time
of writing; the component equivalent of
wasmtime::Val).Static API Oracle
The static API oracle will use the test case generator at build time to
generate, say, 1000 interface types
Tto fuzz, and then generate arbitraryinstances of
Tat runtime. This means that ifTis a struct, for example, wecan define a Rust type like
whose instances can be passed to the Wasm directly via Wasmtime's static API for
interface types. We will generate arbitrary instance of
MyRustTypeat runtimein this scenario, but not new types
T.(Note that
derive(InterfaceType)does not exist at the time of writing.)The text was updated successfully, but these errors were encountered: