-
Notifications
You must be signed in to change notification settings - Fork 110
Fuzzing with AFL
This tutorial walks through the process of fuzzing a Java program using JQF with the AFL fuzzing engine. AFL specializes in fuzzing binary data, and is therefore suited for testing Java programs that parse arbitrary streams of bytes. In particular, JQF-AFL generates random InputStream
objects. If you want to generate test inputs of other types, such as data structures, syntax trees, or any inputs that satisfy specific constraints, use JQF with the Zest engine.
JQF's integration with AFL has been tested on MacOS and Ubuntu, though it should work on any other linux-based systems too. If you have a unix-based system where JQF-AFL does not work as intended, please raise an issue. JQF requires at least AFL version 2.31b to work, though something more recent is preferred (JQF has been tested with AFL 2.52b at the time of this writing).
JQF must be able to find AFL! If you installed afl-fuzz
via standard repositories you may skip this step. JQF will either:
- look for the program
afl-fuzz
in the currentPATH
(typeafl-fuzz
in the command-line to see if it works; also check the minimum version as mentioned above), or - look for the program
afl-fuzz
in the directory specified by the environment variableAFL_DIR
.
The most common way to get started with JQF is to type in bash
:
# Clone and build AFL
git clone https://github.com/google/afl && (cd afl && make)
# Set AFL directory location
export AFL_DIR=$(pwd)/afl
Unlike with other fuzzing engines (e.g. Zest), the use of JQF+AFL requires that you clone the JQF repository and run the set up scripts. The scripts build the JQF-AFL proxy program that communicates signals between the AFL process and the JVM which runs the test program.
# Clone and build JQF
git clone https://github.com/rohanpadhye/jqf && jqf/setup.sh
### See usage of JQF-AFL
jqf/bin/jqf-afl-fuzz
A JQF test driver is a JUnit-style test class with the following annotations: @RunWith(JQF.class)
on the test class, and @Fuzz
on the test method. The test method must be public void
. The formal parameters of the test method are the inputs generated by JQF -- when using JQF+AFL, the test method should have exactly one formal parameter of type InputStream
.
Here is an example of a test driver that tests the in-built PNG image decoding logic in the JDK ImageIO library:
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import edu.berkeley.cs.jqf.fuzz.Fuzz;
import edu.berkeley.cs.jqf.fuzz.JQF;
import org.junit.Assume;
import org.junit.runner.RunWith;
@RunWith(JQF.class)
public class PngTest {
@Fuzz /* JQF will generate inputs to this method */
public void testRead(InputStream input) {
// Create parser
ImageReader reader = ImageIO.getImageReadersByFormatName("png").next();
// Decode image from input stream
try {
reader.setInput(ImageIO.createImageInputStream(input));
// Bound dimensions to avoid OOM
Assume.assumeTrue(reader.getHeight(0) <= 256);
Assume.assumeTrue(reader.getWidth(0) <= 256);
// Decode first image in the input stream
reader.read(0);
} catch (IOException e) {
// This exception signals invalid input and not a test failure
Assume.assumeNoException(e);
}
}
}
Save the above file as PngTest.java. You can use javac
to compile this file as long as you include JQF and all its dependencies on the classpath. JQF ships with a bash script called classpath.sh
which expands to a list of all JARs that it needs. Compile the above file as follows in bash:
javac -cp .:$(jqf/scripts/classpath.sh) PngTest.java # 'jqf' is the directory where JQF is cloned/installed
The program jqf-afl-fuzz
is a wrapper around the program afl-fuzz
that comes with AFL. The wrapper allows you to specify the classpath of your Java tests and then just the test class and method instead of having to invoke the JVM yourself. You can always pass additional options to the JVM (e.g. system properties) using the environment variable JVM_OPTS
.
Usage: jqf-afl-fuzz [options] TEST_CLASS TEST_METHOD
Options:
-c JAVA_CLASSPATH Classpath used to find your test classes (default is '.')
-i AFL_INPUT_DIR Seed inputs for AFL (default is a few seeds of random data)
-o AFL_OUTPUT_DIR Where AFL should save fuzz results (default is './fuzz-results')
-x AFL_DICT Provide a dictionary to AFL (default is no dictionary)
-S WORKER_ID A unique identifier when running in parallel mode
-T AFL_TITLE Customize title banner (default is TEST_CLASS#TEST_METHOD)
-m MEM_LIMIT Set a memory limit in MB (default is 8192)
-t TIMEOUT Set a single-run timeout in milliseconds (default is 10000)
-v Enable verbose logging (in file 'jqf.log')
Let's run jqf-afl-fuzz
on our PNG image decoder, while providing as seed inputs the default PNG files that ship with AFL.
jqf/bin/jqf-afl-fuzz -i $AFL_DIR/testcases/images/png/ PngTest testRead
> Performing pilot run...
The wrapper first performs a pilot run without AFL to ensure that you have set-up your classpath and test driver correctly. If there was any problem, you should see an error message here. Otherwise, if all goes well, you'll see:
> Pilot run success! Launching AFL now...
And then the AFL status screen:
You can reproduce one or more inputs found by AFL using the program jqf-repro
:
jqf/bin/jqf-repro PngTest testRead fuzz-results/crashes/id:000000*
Run the above program without any arguments to see full usage information (e.g. providing custom classpaths).
The repro script will re-execute the test method with the provided inputs, and print exception stacktraces, if any, on the standard error stream.
If the pilot run fails, you should see a stacktrace of the exception that caused this on the screen. Typical reasons for the pilot run failing is providing a wrong or misspelt class name or method name, or providing an incorrect classpath, resulting in a ClassNotFoundException
.
You can also use the -v
option with jqf-afl-fuzz
to make JQF dump logs of certain things in the file jqf.log
, such as the bytecode instrumentation of class files. You will see any errors due to instrumentation in this file, along with anything your test classes write to STDOUT or STDERR.
If you cannot see any errors or the error does not seem to be your fault, please open a GitHub issue.
You can also run multiple instances of JQF-AFL in parallel to utilize multiple CPU cores and speed up fuzzing. Run each sequential instance with the option -S <id>
, where <id>
is some unique identifier for that instance. For example, the following spaws two parallel instances:
jqf/bin/jqf-afl-fuzz -c tests -S id1 PngTest testRead
jqf/bin/jqf-afl-fuzz -c tests -S id2 PngTest testRead
You might have to run the above commands in separate terminals/screens. The output for instance id1
will be in directory fuzz_results/id1
and so on. Remember that you can change the root output directory with the -o
option.
Multi-threaded tests can be difficult to work with. If your test method spawns new threads to do work, make sure to synchronize with these threads before returning (e.g. using Thread.join()
). If a worker thread continues to run after the test method returns, then its code coverage can spill onto subsequent trials (with different inputs) and the fuzzing session will become unreliable.
The source code examples in the wiki pages can be freely re-used under the same license as the rest of JQF.