Merge #30

30: Upgrade libfuzzer for LLVM 6.0 r=frewsxcv a=nagisa

This PR pulls in the changes to libfuzzer, at commit llvm-mirror/compiler-rt@cc0ab3f.

My observation is that this works just fine with current state of affairs. That is, given a:

```
$ rustc -Cllvm-args=-version
LLVM (http://llvm.org/):
  LLVM version 6.0.0
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver1
```

The following command, when run within the both of the `example` folders in this repository work "just" fine (should also be confirmed by the CI), which somewhat counteracts observations made in #29, that libfuzzer cannot work with sanitizers anymore (quite the contrary, it seems that sanitizers are still required).

```
cargo rustc --release -- -Cpasses=sancov -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cpanic=abort -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Zsanitizer=address
```

If I remember correctly, cargo-fuzz generates fuzz targets with this git repository as a dependency. This means that all these fuzz targets will possibly break for people using older compilers once we upgrade, however not upgrading libfuzzer-sys will(?) break people using the new nightly.

While our compatibility story is, I believe, that we support only the current nightly without any real back or forward compatibility, this might be a good time to think about how we want to go about our versioning and release flow. Perhaps we’ll be able to find some way that does not break everybody’s fuzz targets every time LLVMup happens.

Fixes https://github.com/rust-fuzz/libfuzzer-sys/issues/29

r? @frewsxcv or @Manishearth
cc @PaulGrandperrin

.travis.yml

@@ -12,8 +12,8 @@ notifications:
   email: false
 script:
   - cd example
-  - cargo rustc --release -- -C passes='sancov' -C llvm-args='-sanitizer-coverage-level=4' -Z sanitizer=address
+  - cargo rustc --release -- -Cpasses='sancov' -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cpanic=abort -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Zsanitizer=address
   - (! ./target/release/example -runs=100000)
   - cd ../example_arbitrary
-  - cargo rustc --release -- -C passes='sancov' -C llvm-args='-sanitizer-coverage-level=4' -Z sanitizer=address
+  - cargo rustc --release -- -Cpasses='sancov' -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cpanic=abort -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Zsanitizer=address
   - (! ./target/release/example -runs=10000000)

README.md

@@ -1,6 +1,6 @@
 Barebones wrapper around libFuzzer runtime library.
 
-The CPP parts are extracted from llvm git repository with `git filter-branch`.
+The CPP parts are extracted from compiler-rt git repository with `git filter-branch`.
 
 libFuzzer relies on LLVM sanitizer support. The Rust compiler has built-in support for LLVM sanitizer support, for now, it's limited to Linux. As a result, libfuzzer-sys only works on Linux.
 
@@ -50,4 +50,5 @@ $ ./target/debug/fuzzed # runs fuzzing
 
 ## License
 
-All files in `llvm` are licensed NCSA. Everything else is dual-licensed Apache 2.0 and MIT.
+All files in `libfuzzer` directory are licensed NCSA.
+Everything else is dual-licensed Apache 2.0 and MIT.

build.rs

@@ -14,7 +14,7 @@ fn main() {
         println!("cargo:rustc-link-lib=stdc++");
     } else {
         let mut build = cc::Build::new();
-        let sources = ::std::fs::read_dir("llvm/lib/Fuzzer")
+        let sources = ::std::fs::read_dir("libfuzzer")
             .expect("listable source directory")
             .map(|de| de.expect("file in directory").path())
             .filter(|p| p.extension().map(|ext| ext == "cpp") == Some(true))
@@ -24,6 +24,7 @@ fn main() {
         }
         build.flag("-std=c++11");
         build.flag("-fno-omit-frame-pointer");
+        build.flag("-w");
         build.cpp(true);
         build.compile("libfuzzer.a");
     }

libfuzzer/CMakeLists.txt

@@ -0,0 +1,118 @@
+set(LIBFUZZER_SOURCES
+  FuzzerClangCounters.cpp
+  FuzzerCrossOver.cpp
+  FuzzerDriver.cpp
+  FuzzerExtFunctionsDlsym.cpp
+  FuzzerExtFunctionsDlsymWin.cpp
+  FuzzerExtFunctionsWeak.cpp
+  FuzzerExtraCounters.cpp
+  FuzzerIO.cpp
+  FuzzerIOPosix.cpp
+  FuzzerIOWindows.cpp
+  FuzzerLoop.cpp
+  FuzzerMerge.cpp
+  FuzzerMutate.cpp
+  FuzzerSHA1.cpp
+  FuzzerShmemPosix.cpp
+  FuzzerShmemWindows.cpp
+  FuzzerTracePC.cpp
+  FuzzerUtil.cpp
+  FuzzerUtilDarwin.cpp
+  FuzzerUtilFuchsia.cpp
+  FuzzerUtilLinux.cpp
+  FuzzerUtilPosix.cpp
+  FuzzerUtilWindows.cpp
+  )
+
+CHECK_CXX_SOURCE_COMPILES("
+  static thread_local int blah;
+  int main() {
+  return 0;
+  }
+  " HAS_THREAD_LOCAL)
+
+set(LIBFUZZER_CFLAGS ${SANITIZER_COMMON_CFLAGS})
+
+if("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux" AND COMPILER_RT_LIBCXX_PATH)
+  list(APPEND LIBFUZZER_CFLAGS -nostdinc++ -D_LIBCPP_ABI_VERSION=Fuzzer)
+  # Remove -stdlib= which is unused when passing -nostdinc++.
+  string(REGEX REPLACE "-stdlib=[a-zA-Z+]*" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
+endif()
+
+append_list_if(COMPILER_RT_HAS_OMIT_FRAME_POINTER_FLAG -fno-omit-frame-pointer LIBFUZZER_CFLAGS)
+
+if (CMAKE_CXX_FLAGS MATCHES "fsanitize-coverage")
+  list(APPEND LIBFUZZER_CFLAGS -fno-sanitize-coverage=trace-pc-guard,edge,trace-cmp,indirect-calls,8bit-counters)
+endif()
+
+if(NOT HAS_THREAD_LOCAL)
+  list(APPEND LIBFUZZER_CFLAGS -Dthread_local=__thread)
+endif()
+
+if(APPLE)
+  set(FUZZER_SUPPORTED_OS osx)
+endif()
+
+add_compiler_rt_object_libraries(RTfuzzer
+  OS ${FUZZER_SUPPORTED_OS}
+  ARCHS ${FUZZER_SUPPORTED_ARCH}
+  SOURCES ${LIBFUZZER_SOURCES}
+  CFLAGS ${LIBFUZZER_CFLAGS})
+
+add_compiler_rt_object_libraries(RTfuzzer_main
+  OS ${FUZZER_SUPPORTED_OS}
+  ARCHS ${FUZZER_SUPPORTED_ARCH}
+  SOURCES FuzzerMain.cpp
+  CFLAGS ${LIBFUZZER_CFLAGS})
+
+add_compiler_rt_runtime(clang_rt.fuzzer
+  STATIC
+  OS ${FUZZER_SUPPORTED_OS}
+  ARCHS ${FUZZER_SUPPORTED_ARCH}
+  OBJECT_LIBS RTfuzzer RTfuzzer_main
+  CFLAGS ${LIBFUZZER_CFLAGS}
+  PARENT_TARGET fuzzer)
+
+add_compiler_rt_runtime(clang_rt.fuzzer_no_main
+  STATIC
+  OS ${FUZZER_SUPPORTED_OS}
+  ARCHS ${FUZZER_SUPPORTED_ARCH}
+  OBJECT_LIBS RTfuzzer
+  CFLAGS ${LIBFUZZER_CFLAGS}
+  PARENT_TARGET fuzzer)
+
+if("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux" AND COMPILER_RT_LIBCXX_PATH)
+  macro(partially_link_libcxx name dir arch)
+    set(cxx_${arch}_merge_dir "${CMAKE_CURRENT_BINARY_DIR}/cxx_${arch}_merge.dir")
+    file(MAKE_DIRECTORY ${cxx_${arch}_merge_dir})
+    add_custom_command(TARGET clang_rt.${name}-${arch} POST_BUILD
+      COMMAND ${CMAKE_LINKER} --whole-archive "$<TARGET_LINKER_FILE:clang_rt.${name}-${arch}>" --no-whole-archive ${dir}/src/libcxx_fuzzer_${arch}-build/lib/libc++.a -r -o ${name}.o
+      COMMAND ${CMAKE_OBJCOPY} --localize-hidden ${name}.o
+      COMMAND ${CMAKE_COMMAND} -E remove "$<TARGET_LINKER_FILE:clang_rt.${name}-${arch}>"
+      COMMAND ${CMAKE_AR} qcs "$<TARGET_LINKER_FILE:clang_rt.${name}-${arch}>" ${name}.o
+      WORKING_DIRECTORY ${cxx_${arch}_merge_dir}
+    )
+  endmacro()
+
+  foreach(arch ${FUZZER_SUPPORTED_ARCH})
+    get_target_flags_for_arch(${arch} TARGET_CFLAGS)
+    set(LIBCXX_${arch}_PREFIX ${CMAKE_CURRENT_BINARY_DIR}/libcxx_fuzzer_${arch})
+    add_custom_libcxx(libcxx_fuzzer_${arch} ${LIBCXX_${arch}_PREFIX}
+      CFLAGS ${TARGET_CFLAGS}
+             -D_LIBCPP_ABI_VERSION=Fuzzer
+             -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS=1
+             -fvisibility=hidden
+      CMAKE_ARGS -DLIBCXX_ENABLE_EXCEPTIONS=OFF
+                 -DLIBCXX_CXX_ABI=none)
+    target_compile_options(RTfuzzer.${arch} PRIVATE -isystem ${COMPILER_RT_LIBCXX_PATH}/include)
+    add_dependencies(RTfuzzer.${arch} libcxx_fuzzer_${arch}-build)
+    target_compile_options(RTfuzzer_main.${arch} PRIVATE -isystem ${COMPILER_RT_LIBCXX_PATH}/include)
+    add_dependencies(RTfuzzer_main.${arch} libcxx_fuzzer_${arch}-build)
+    partially_link_libcxx(fuzzer_no_main ${LIBCXX_${arch}_PREFIX} ${arch})
+    partially_link_libcxx(fuzzer ${LIBCXX_${arch}_PREFIX} ${arch})
+  endforeach()
+endif()
+
+if(COMPILER_RT_INCLUDE_TESTS)
+  add_subdirectory(tests)
+endif()

libfuzzer/FuzzerClangCounters.cpp

@@ -0,0 +1,49 @@
+//===- FuzzerExtraCounters.cpp - Extra coverage counters ------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+// Coverage counters from Clang's SourceBasedCodeCoverage.
+//===----------------------------------------------------------------------===//
+
+// Support for SourceBasedCodeCoverage is experimental:
+// * Works only for the main binary, not DSOs yet.
+// * Works only on Linux.
+// * Does not implement print_pcs/print_coverage yet.
+// * Is not fully evaluated for performance and sensitivity.
+//   We expect large performance drop due to 64-bit counters,
+//   and *maybe* better sensitivity due to more fine-grained counters.
+//   Preliminary comparison on a single benchmark (RE2) shows
+//   a bit worse sensitivity though.
+
+#include "FuzzerDefs.h"
+
+#if LIBFUZZER_LINUX
+__attribute__((weak)) extern uint64_t __start___llvm_prf_cnts;
+__attribute__((weak)) extern uint64_t __stop___llvm_prf_cnts;
+namespace fuzzer {
+uint64_t *ClangCountersBegin() { return &__start___llvm_prf_cnts; }
+uint64_t *ClangCountersEnd() { return &__stop___llvm_prf_cnts; }
+}  // namespace fuzzer
+#else
+// TODO: Implement on Mac (if the data shows it's worth it).
+//__attribute__((visibility("hidden")))
+//extern uint64_t CountersStart __asm("section$start$__DATA$__llvm_prf_cnts");
+//__attribute__((visibility("hidden")))
+//extern uint64_t CountersEnd __asm("section$end$__DATA$__llvm_prf_cnts");
+namespace fuzzer {
+uint64_t *ClangCountersBegin() { return nullptr; }
+uint64_t *ClangCountersEnd() { return  nullptr; }
+}  // namespace fuzzer
+#endif
+
+namespace fuzzer {
+ATTRIBUTE_NO_SANITIZE_ALL
+void ClearClangCounters() {  // hand-written memset, don't asan-ify.
+  for (auto P = ClangCountersBegin(); P < ClangCountersEnd(); P++)
+    *P = 0;
+}
+}

libfuzzer/FuzzerCommand.h

@@ -0,0 +1,180 @@
+//===- FuzzerCommand.h - Interface representing a process -------*- C++ -* ===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+// FuzzerCommand represents a command to run in a subprocess.  It allows callers
+// to manage command line arguments and output and error streams.
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_FUZZER_COMMAND_H
+#define LLVM_FUZZER_COMMAND_H
+
+#include "FuzzerDefs.h"
+#include "FuzzerIO.h"
+
+#include <algorithm>
+#include <sstream>
+#include <string>
+#include <vector>
+
+namespace fuzzer {
+
+class Command final {
+public:
+  // This command line flag is used to indicate that the remaining command line
+  // is immutable, meaning this flag effectively marks the end of the mutable
+  // argument list.
+  static inline const char *ignoreRemainingArgs() {
+    static const char *kIgnoreRemaining = "-ignore_remaining_args=1";
+    return kIgnoreRemaining;
+  }
+
+  Command() : CombinedOutAndErr(false) {}
+
+  explicit Command(const Vector<std::string> &ArgsToAdd)
+      : Args(ArgsToAdd), CombinedOutAndErr(false) {}
+
+  explicit Command(const Command &Other)
+      : Args(Other.Args), CombinedOutAndErr(Other.CombinedOutAndErr),
+        OutputFile(Other.OutputFile) {}
+
+  Command &operator=(const Command &Other) {
+    Args = Other.Args;
+    CombinedOutAndErr = Other.CombinedOutAndErr;
+    OutputFile = Other.OutputFile;
+    return *this;
+  }
+
+  ~Command() {}
+
+  // Returns true if the given Arg is present in Args.  Only checks up to
+  // "-ignore_remaining_args=1".
+  bool hasArgument(const std::string &Arg) const {
+    auto i = endMutableArgs();
+    return std::find(Args.begin(), i, Arg) != i;
+  }
+
+  // Gets all of the current command line arguments, **including** those after
+  // "-ignore-remaining-args=1".
+  const Vector<std::string> &getArguments() const { return Args; }
+
+  // Adds the given argument before "-ignore_remaining_args=1", or at the end
+  // if that flag isn't present.
+  void addArgument(const std::string &Arg) {
+    Args.insert(endMutableArgs(), Arg);
+  }
+
+  // Adds all given arguments before "-ignore_remaining_args=1", or at the end
+  // if that flag isn't present.
+  void addArguments(const Vector<std::string> &ArgsToAdd) {
+    Args.insert(endMutableArgs(), ArgsToAdd.begin(), ArgsToAdd.end());
+  }
+
+  // Removes the given argument from the command argument list.  Ignores any
+  // occurrences after "-ignore_remaining_args=1", if present.
+  void removeArgument(const std::string &Arg) {
+    auto i = endMutableArgs();
+    Args.erase(std::remove(Args.begin(), i, Arg), i);
+  }
+
+  // Like hasArgument, but checks for "-[Flag]=...".
+  bool hasFlag(const std::string &Flag) {
+    std::string Arg("-" + Flag + "=");
+    auto IsMatch = [&](const std::string &Other) {
+      return Arg.compare(0, std::string::npos, Other, 0, Arg.length()) == 0;
+    };
+    return std::any_of(Args.begin(), endMutableArgs(), IsMatch);
+  }
+
+  // Returns the value of the first instance of a given flag, or an empty string
+  // if the flag isn't present.  Ignores any occurrences after
+  // "-ignore_remaining_args=1", if present.
+  std::string getFlagValue(const std::string &Flag) {
+    std::string Arg("-" + Flag + "=");
+    auto IsMatch = [&](const std::string &Other) {
+      return Arg.compare(0, std::string::npos, Other, 0, Arg.length()) == 0;
+    };
+    auto i = endMutableArgs();
+    auto j = std::find_if(Args.begin(), i, IsMatch);
+    std::string result;
+    if (j != i) {
+      result = j->substr(Arg.length());
+    }
+    return result;
+  }
+
+  // Like AddArgument, but adds "-[Flag]=[Value]".
+  void addFlag(const std::string &Flag, const std::string &Value) {
+    addArgument("-" + Flag + "=" + Value);
+  }
+
+  // Like RemoveArgument, but removes "-[Flag]=...".
+  void removeFlag(const std::string &Flag) {
+    std::string Arg("-" + Flag + "=");
+    auto IsMatch = [&](const std::string &Other) {
+      return Arg.compare(0, std::string::npos, Other, 0, Arg.length()) == 0;
+    };
+    auto i = endMutableArgs();
+    Args.erase(std::remove_if(Args.begin(), i, IsMatch), i);
+  }
+
+  // Returns whether the command's stdout is being written to an output file.
+  bool hasOutputFile() const { return !OutputFile.empty(); }
+
+  // Returns the currently set output file.
+  const std::string &getOutputFile() const { return OutputFile; }
+
+  // Configures the command to redirect its output to the name file.
+  void setOutputFile(const std::string &FileName) { OutputFile = FileName; }
+
+  // Returns whether the command's stderr is redirected to stdout.
+  bool isOutAndErrCombined() const { return CombinedOutAndErr; }
+
+  // Sets whether to redirect the command's stderr to its stdout.
+  void combineOutAndErr(bool combine = true) { CombinedOutAndErr = combine; }
+
+  // Returns a string representation of the command.  On many systems this will
+  // be the equivalent command line.
+  std::string toString() const {
+    std::stringstream SS;
+    for (auto arg : getArguments())
+      SS << arg << " ";
+    if (hasOutputFile())
+      SS << ">" << getOutputFile() << " ";
+    if (isOutAndErrCombined())
+      SS << "2>&1 ";
+    std::string result = SS.str();
+    if (!result.empty())
+      result = result.substr(0, result.length() - 1);
+    return result;
+  }
+
+private:
+  Command(Command &&Other) = delete;
+  Command &operator=(Command &&Other) = delete;
+
+  Vector<std::string>::iterator endMutableArgs() {
+    return std::find(Args.begin(), Args.end(), ignoreRemainingArgs());
+  }
+
+  Vector<std::string>::const_iterator endMutableArgs() const {
+    return std::find(Args.begin(), Args.end(), ignoreRemainingArgs());
+  }
+
+  // The command arguments.  Args[0] is the command name.
+  Vector<std::string> Args;
+
+  // True indicates stderr is redirected to stdout.
+  bool CombinedOutAndErr;
+
+  // If not empty, stdout is redirected to the named file.
+  std::string OutputFile;
+};
+
+} // namespace fuzzer
+
+#endif // LLVM_FUZZER_COMMAND_H