Fuzzing with american fuzzy lop

[ilammy]

This one has been on my mind for quite a while and finally I've managed to get it going. Let's throw a security fuzzer into the breach and see what it finds for us. (Oh boy, some crashes does it find! I've got seven of them reported in seven minutes. Based on a single input file with no other hints.)

Here we add american fuzzy lop because I like how little tweaking and configuration it requires. Basically, you feed it an example input data and it uses — high technology from 1960s — artificial intelligence (!) and machine learning (!!) to work through big data (!!!) queue of tests trying to invent ones that crash your application.

You can read user manual (of a sort) in the README. In order to run the fuzzer you need to install it first (apt get afl or similar) and then do make fuzz FUZZ_BIN=something to build and run the tools.

Implementation-wise, we make a custom build of Themis (by recursively calling make because that's the easiest way) which is instrumented by a special compiler. This allows the fuzzer to monitor the behavior of Themis and the tools and see how the input influences the control flow in the program.

Only two tools are implemented for starters:

• Round-trip through a Secure Cell in sealing mode.

  This makes sure that user input cannot be used to produce a secure container which crashes the application during processing.

• Decrypting a container with Secure Cell in sealing mode.

  This makes sure that data corruption cannot cause the application to crash when receiving messages encrypted with Secure Cell from untrusted sources.

It should be easy to add more tools in the future. For example, Secure Message could use a fuzzer for key files. Other components use the same containers as Secure Cell so fuzzing the encrypted data may not be so fruitful, but it may still be worth a shot.

I admit that error handling and memory management in the tools is a bit sloppy but that should be fine unless it crashes in unexpected places. It's just too verbose to do everything right.

Finally, we add a make fuzz step to the CI build in order to keep up with API changes. We don't run the fuzzing automatically but let's at least ensure that the tools can be compiled. (We can also check that they can handle the input data but it's not that important.)

[vixentael]

Wow, that's really cool!
I'll play with it too :)

I believe in future we might want to add fuzzer tests in CI as well (depends on false positives number).

[vixentael]

Do you find way to investigate crash results?

I found that afl has crash triage tool for pointing out the crash source, but it's written for GDB (I have LLDB), and I haven't found an easy way to point LLDB to the crash.

[gene-eu-zz]

In fact, we've manually fuzzed Themis until some version, but results and methodology were rather obscure (it fruited in some bugs we've removed, but last 2 years we barely did it due to limited changes in Themis core). But if we can come up with plausible automatic scheme and add it to CI, my heart will beat calmer.

[ilammy]

I believe in future we might want to add fuzzer tests in CI as well (depends on false positives number).

Automation raises a whole bunch of new questions. For example, how are we going to collect the results? We'll need to salvage the test inputs with interesting behavior, get them out of the build box somehow, etc. Then there's resource allocation where fuzzing may find something interesting, but this would require a couple of hours of torture tests or something. There's little value in shallow tests that can be run in five minutes but do not yield any interesting results in years.

I agree that this is an interesting topic to pursue though.

Do you find way to investigate crash results?

I've reproduced the crashes manually and saved the input data for further investigation. I haven't looked into the cause deeply yet other than running it under debugger and confirming that the crashes are caused by some OpenSSL functions.

I have seen a bunch of helper tools to deal with reporting automatically, but did not look too deeply into that yet as well. For now the developers will have to manually pick up the tool binary from build directory, pipe the failing test case into it, and observe the results. I imagine it can be automated.