Fuzz testing with cargo fuzz

[LegNeato]

We should fuzz test execute and execute_async to make sure no external input can cause the library to panic or crash.

[trevyn]

I set up a basic cargo fuzz for execute_sync here: trevyn/juniper@master...trevyn:fuzz

It got up to 1.8M executions (~20 minutes) and then segfaulted:

#1842766        REDUCE cov: 2716 ft: 10646 corp: 1777/747Kb lim: 3205 exec/s: 1360 rss: 830Mb L: 17/3073 MS: 1 EraseBytes-
────────────────────────────────────────────────────────────────────────────────

Error: Fuzz target exited with signal: 11

Apparently there is a known bug in cargo fuzz where it doesn't save the test case, and one way to find it is to use afl.rs🤪: rust-fuzz/cargo-fuzz#235 (comment)

At any rate, it seems like what's really getting exercised is parse_document_source here: https://docs.rs/juniper/0.15.3/src/juniper/lib.rs.html#231

@LegNeato is that as intended? Also, I'm not sure if the async-flavored execute function is exercising anything different, and I'd have to figure out how to scaffold that.

[trevyn]

Ok, here's the AFL version: trevyn/juniper@master...trevyn:fuzz-afl

Ran it for 30 minutes with no issues, will leave it running and report back later.

[trevyn]

1 crash found: crash-0000.min

I added a (failing) test to the trevyn:fuzz-afl branch.

[trevyn]

In addition to the stack overflow crash above, it's also easy to hit a Type not found in schema panic. That file has a lot of expect()s that look easy to hit, though, so I'm not sure what the intended behavior is.

[LegNeato]

Woah, totally missed this, great work! Is it ready to be integrated?

[LegNeato]

@trevyn , FWIW I was thinking about this due to 17d474e.

I wonder if there is a way to give a schema and have a bunch of valid and invalid permutations thrown at us. Would be useful for all GraphQL servers to have a corpus of graphql queries with corresponding schema and if they should pass / not pass.

[trevyn]

@LegNeato Thanks! I’m not working with Juniper or GraphQL anymore, but feel free to use anything here!