Exercise fuzzers during CI builds #579
Conversation
Instead of using our special AFL_CFLAGS and AFL_LDFLAGS, just set them per-target as we do in other places. We'd like to build AFL fuzzers with sanitizers, and we'd like to be aware when AFL toolchain does not support a particular compilation flag. Currently we can WITH_ASAN=1, but it silently drops sanitizer flags instead of complaining, which gives an impression that we use sanitizer when in fact we do not.
Pull in Soter dependencies when building Themis so that it triggers BoringSSL build when ENGINE is set.
AFL does not support 64-bit builds so build 32-bit binaries instead. Note that we have to do in two places: during original make invocation to build fuzzer tools, and during recursive make invocation to build Themis library with AFL instrumentation.
It turns out that afl-gcc actually uses AFL_CC environment variable (who would have though, eh?) And if we set it to afl-gcc then afl-gcc tries to invoke afl-gcc in an infinite recursive loop. Avoid that by unexporting the variable, keeping it only inside make.
Introduce a new "fuzz" job that builds, tests, and runs fuzzers.
We check that we can build all interesting configurations and run
the most interesting ones for a bit to ensure that there are no
obvious mistakes in the code. We'd like to run longer, but CircleCI
has around 20-minute limit which does not fit what we want to eat.
AFL is meant to be used interactively (primarily) but it is possible
to script its execution too. We use a bit peculiar spell for that:
timeout -s INT 30s make fuzz | cat -u || true
with the following rationale:
- timeout limits execution time of "make fuzz" which is by default
unbounded. Put a 30-second limit on execution to keep it civil.
We've got quite a few fuzzing tools to run.
- AFL expects to be cleanly stopped by SIGINT. However, it will stop
with non-zero exit code, so we eat it with "|| true"
- Piping into cat disconnects AFL from terminal so it won't be
screaming about "terminal window is too small, plz resize it
to at least 80x25, kthx"). Everything is better with cats.
CircleCI runs stuff in unprivileged Docker containers and gdb requires ptrace capabilities to attach to processes. Unfortunately, we'll get no stack traces on CI. Well, add an option to skip debugger analysis and just try crashing, maybe that will output something interesting.
Often fuzzing finds some length field, inverts it, and this results in a gigabyte-sized allocation that malloc() refuses to fulfill. However, it does not set "errno" which we use for error reporting at call site resulting in silly errors: "failed to read file.dat: Success". Let's set errno to something sensible before exiting. Also, limit the input buffer sizes to 50 MB. AFL can bit-flip those lengths too which is not what we really want. Since we use 32-bit lengths those are mostly fine on 64-bit systems, but when running 32-bit binaries with ASAN malloc() tends to panic when allocating more than 2 GB. Don't let it do that and abort the test (cleanly).
We don't have direct access to reports on CI so let's dump the input into logs so that it's readily accessible. It's usually something small so it's okay to write it out.
It is possible for report directory to contain multiple reports, with some not having crashes. "print_banner" gets reset for each report set so we cannot use. Introduce a dedicated flag to note that we have had a crash and should report that.
ilammy
added
core
| #include <stdlib.h> | ||
|
|
||
| #define MAX_SANE_LENGTH (50 * 1024 * 1024) |
There was a problem hiding this comment.
Yeah, 50 MB. This is default memory limit of AFL and our test data is unlikely to exceed it. There is no gain in making a huge test case as AFL will try minimizing it anyway.
|
So,
|
YAML programmers 🤣
They are running in parallel, and most of the time Android builds and integration tests are the slowest, so we're unlikely to be blocked by this. (And it will be about 2 minutes and counting in fact: there are 2 fuzzing tools and 2 fuzzing modes, each of those 4 runs for 30 seconds.)
That's correct.
I guess manually inspect the failure and use our best judgement. I'd probably rerun the job, if the failure persists and is definitely not related to PR, they we might note the failure, approve the merge with it, and fix it later separately. Best practice would be to run shadow builds and send alerts about fuzzing failures only to maintainers, without exposing PR authors to them. But I'm not really sure how this can be easily done with CircleCI. I did not notice any ready-made so we'd have to hack something (e.g., note the failure, send an alert via API integration, let the build succeed if the failure is found by fuzzers). |
We'd been building 32-bit binaries when ASAN was enabled even withou AFL. ASAN works fine on general 64-bit systems, we need to build 32-bit binaries only when running AFL. Use -m32 when building fuzzing tools. Variable adjuments when AFL_USE_ASAN=1 will take care of the modified Themis build.
does fuzzer can save test data to some file? if so we can save it as test artifacts to reproduce it later on failures |
Yes. The results are saved in the build tree, along with all the intermediate data. We use this data to generate reports (which are dumped into build logs). There should be enough data in the logs to reproduce the crashes:
We do not save the original report data since I thought all of the above should be sufficient. But it should be easy to store full reports on CircleCI as builds artifacts. Who's for it? |
In addition to dumping information to log, save full reports in CircleCI artifacts for each build so that they may be retrieved later. There are quite a few small files in there so zip them into a single archive.
|
In my perfect world (🦄) we'd have some artifact (logs in file, result of AFL) that we can grab from CI and post into internal issue tracker. So, smth that can be downloaded and attached. Artifacts should work i think |
Yeah, okay, git, I get it, you're great at merge conflicts.
I mean, resolve merge conflicts because evidently I can't resolve them on the first attempt.
|
PR #592 contains fixes which should resolve issues found by AFL in this PR. Once that is merged, master should be merged into this PR and the build is expected to go green. |
We have had some kind of a fuzzing framework for close to a year, but it did not get enough attention since only nobody cares about that. Let’s improve the situation a bit.
Make sure that fuzzers can be built with sanitizers. This is the most interesting case since sanitizers report bugs by crashing and AFL reacts to crashes. Building and running ASAN tests was a quest on its own.
Build and run fuzzers on CI. Every time. For 30 seconds each. Hopefully we’ll catch some low-hanging bugs with that, given a sufficient number of builds.
There is at least one known bug which is caught by fuzzers+sanitizers, so hopefully the build will fail this time and you can see the report.
Improve reporting and crash analysis tool so that they are more useful.
There are more implementation details in commit messages.
P.S. Yep, the build does fail right now.
Failures are also stored in artifacts, example.
Checklist
Changelog is updated(maybe later)