[fuzz] add fuzz coverage CI check by asraa · Pull Request #11045 · envoyproxy/envoy

asraa · 2020-05-04T18:34:51Z

Commit Message: Add bazel.fuzz_coverage to CI. Publishes fuzz coverage report in coverage_publish
Risk Level: Medium
Testing: Local testing of ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.fuzz_coverage', as well as FUZZ_COVERAGE=true test/run_envoy_bazel_coverage.sh with single test args/directory, VALIDATE on/off.
Fixes: #9573
Docs: Updated bazel/README.md and fuzz/README.md for docs about running local coverage.

Signed-off-by: Asra Ali asraa@google.com

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-05-04T18:36:57Z

@htuch @lizan This is my first stab at adding to CI. coverage_publish publishes both reports, and fuzz coverage writes out to /generated/fuzz_coverage now. I made the fuzz coverage at 27% right now, hoping to increase this as we see gaps/add fuzz targets over Q2/Q3.

Signed-off-by: Asra Ali <asraa@google.com>

htuch · 2020-05-05T13:46:07Z

@asraa Very cool. LMK when this passes; I'd like to take a look at the artifacts in CI. Are you planning on adding a gating check on overall fuzz coverage to encourage folks to write more fuzzers in this PR?

htuch · 2020-05-05T13:46:11Z

/wait

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-05-08T19:33:19Z

CI failure is interesting. An easy solution is to run the gtest mode of the fuzz target when running bazel coverage.

I pushed the server corpus entry that triggers the ApiBoostRetryException and failure to the PR. It crashes with an uncaught exception. Reproduce with:
bazel run --config=libc++ --test_output=all --test_arg=/usr/local/google/home/asraa/git/envoy2/test/server/server_corpus --test_arg=-runs=0 //test/server/config_validation:config_fuzz_test_with_libfuzzer
(Replace with your own srcdir). This runs the test against the seed corpus with the fuzzing driver.

It is important to run the target _with_libfuzzer that links the fuzzing main. It does not repro with using standard gtest main.
It must use --config=libc++. Without it, it succeeds.
If you add --config clang-asan in addition to --config=libc++, it will succeed.
There are some (DO NOT MERGE) std::cerr logs I added so that they can be seen in fuzzing mode. It indeed is getting stuck in applying the lambda.

@yanavlasov Does this indicate that this might be a compiler bug?
stack trace shows the exception is uncaught in tryWithApiBoosting executing the lambda here:

terminating with uncaught exception of type Envoy::(anonymous namespace)::ApiBoostRetryException
==36646== ERROR: libFuzzer: deadly signal
    #0 0x2e75450 in __sanitizer_print_stack_trace (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x2e75450)
    #1 0x2e20818 in fuzzer::PrintStackTrace() (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x2e20818)
    #2 0x2e05f83 in fuzzer::Fuzzer::CrashCallback() (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x2e05f83)
    #3 0x7f8b9648f38f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
    #4 0x7f8b960e9427 in raise (/lib/x86_64-linux-gnu/libc.so.6+0x35427)
    #5 0x7f8b960eb029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029)
    #6 0x8aafb0a in abort_message (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aafb0a)
    #7 0x8aafa35 in demangling_terminate_handler() (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aafa35)
    #8 0x8aaf892 in std::__terminate(void (*)()) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aaf892)
    #9 0x8aaeee5 in __cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aaeee5)
    #10 0x8aaee7e in __cxa_throw (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aaee7e)
    #11 0x72d19a6 in Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion) const (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d19a6)
    #12 0x72e25a9 in decltype(std::__1::forward<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&>(fp)(std::__1::forward<google::protobuf::Message&>(fp0), std::__1::forward<Envoy::(anonymous namespace)::MessageVersion>(fp0))) std::__1::__invoke<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion>(Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e25a9)
    #13 0x72e24fa in void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion>(Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e24fa)
    #14 0x72e248a in std::__1::__function::__alloc_func<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1, std::__1::allocator<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1>, void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e248a)
    #15 0x72e0e6c in std::__1::__function::__func<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1, std::__1::allocator<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1>, void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e0e6c)
    #16 0x72d4d02 in std::__1::__function::__value_func<void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) const (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d4d02)
    #17 0x72d4c67 in std::__1::function<void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion) const (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d4c67)
    #18 0x72d00bb in Envoy::(anonymous namespace)::tryWithApiBoosting(std::__1::function<void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>, google::protobuf::Message&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d00bb)

Note: There are a lot of lambas in this stack... loadBootstrapConfig passes a lambda to loadBootstrap that executes loadFromFile. Then loadFromFile passes a lambda that throws to tryWithApiBoostring that encapsulates the lambda with a try/catch.

Signed-off-by: Asra Ali <asraa@google.com>

lizan · 2020-05-08T22:13:26Z

yeah I think we might want to hold this until we sort out multi-binary coverage #10909, I think that will get rid of most issues we hit with coverage.

asraa · 2020-05-11T12:32:29Z

marking as draft, for both the clang issue and multibinary coverage

stale · 2020-05-19T06:14:29Z

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-05-21T13:43:11Z

This is ready for review!

Edit: Coverage report produced https://347756-65214191-gh.circle-artifacts.com/0/fuzz_coverage/index.html

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-05-21T18:24:01Z

:/ There's a regression from a recent PR submitted. Will ping when it's resolved.

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-05-26T14:27:06Z

H/2 test timeout is blocking CI: #11290. Will merge master when fix is merged

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-06-25T13:15:04Z

Fuzz CI passes when I build with size optimization (ASSERTs are still caught)

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-07-01T20:52:43Z

CI failure is interesting. An easy solution is to run the gtest mode of the fuzz target when running bazel coverage.

I pushed the server corpus entry that triggers the ApiBoostRetryException and failure to the PR. It crashes with an uncaught exception. Reproduce with:
bazel run --config=libc++ --test_output=all --test_arg=/usr/local/google/home/asraa/git/envoy2/test/server/server_corpus --test_arg=-runs=0 //test/server/config_validation:config_fuzz_test_with_libfuzzer
(Replace with your own srcdir). This runs the test against the seed corpus with the fuzzing driver.

It is important to run the target _with_libfuzzer that links the fuzzing main. It does not repro with using standard gtest main.
It must use --config=libc++. Without it, it succeeds.
If you add --config clang-asan in addition to --config=libc++, it will succeed.
There are some (DO NOT MERGE) std::cerr logs I added so that they can be seen in fuzzing mode. It indeed is getting stuck in applying the lambda.

@yanavlasov Does this indicate that this might be a compiler bug?
stack trace shows the exception is uncaught in tryWithApiBoosting executing the lambda here:

terminating with uncaught exception of type Envoy::(anonymous namespace)::ApiBoostRetryException
==36646== ERROR: libFuzzer: deadly signal
    #0 0x2e75450 in __sanitizer_print_stack_trace (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x2e75450)
    #1 0x2e20818 in fuzzer::PrintStackTrace() (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x2e20818)
    #2 0x2e05f83 in fuzzer::Fuzzer::CrashCallback() (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x2e05f83)
    #3 0x7f8b9648f38f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
    #4 0x7f8b960e9427 in raise (/lib/x86_64-linux-gnu/libc.so.6+0x35427)
    #5 0x7f8b960eb029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029)
    #6 0x8aafb0a in abort_message (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aafb0a)
    #7 0x8aafa35 in demangling_terminate_handler() (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aafa35)
    #8 0x8aaf892 in std::__terminate(void (*)()) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aaf892)
    #9 0x8aaeee5 in __cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aaeee5)
    #10 0x8aaee7e in __cxa_throw (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x8aaee7e)
    #11 0x72d19a6 in Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion) const (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d19a6)
    #12 0x72e25a9 in decltype(std::__1::forward<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&>(fp)(std::__1::forward<google::protobuf::Message&>(fp0), std::__1::forward<Envoy::(anonymous namespace)::MessageVersion>(fp0))) std::__1::__invoke<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion>(Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e25a9)
    #13 0x72e24fa in void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion>(Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1&, google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e24fa)
    #14 0x72e248a in std::__1::__function::__alloc_func<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1, std::__1::allocator<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1>, void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e248a)
    #15 0x72e0e6c in std::__1::__function::__func<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1, std::__1::allocator<Envoy::MessageUtil::loadFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Message&, Envoy::ProtobufMessage::ValidationVisitor&, Envoy::Api::Api&, bool)::$_1>, void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72e0e6c)
    #16 0x72d4d02 in std::__1::__function::__value_func<void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion&&) const (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d4d02)
    #17 0x72d4c67 in std::__1::function<void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>::operator()(google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion) const (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d4c67)
    #18 0x72d00bb in Envoy::(anonymous namespace)::tryWithApiBoosting(std::__1::function<void (google::protobuf::Message&, Envoy::(anonymous namespace)::MessageVersion)>, google::protobuf::Message&) (/build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-fastbuild/bin/test/server/config_validation/config_fuzz_test_with_libfuzzer+0x72d00bb)

Note: There are a lot of lambas in this stack... loadBootstrapConfig passes a lambda to loadBootstrap that executes loadFromFile. Then loadFromFile passes a lambda that throws to tryWithApiBoostring that encapsulates the lambda with a try/catch.

asraa · 2020-07-01T20:54:44Z

The close was by accident. It turns out interns are also running into this issue locally #11802

I got coverage without asan working, so i'll make that fix a separate PR and reopen this when when it's ready.

Signed-off-by: Asra Ali <asraa@google.com>

test/run_envoy_bazel_coverage.sh

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2020-07-16T19:32:44Z

Finally, success.

using fuzzer-no-link in bazelrc cli breaks the gtest that's used to build the corpus, hence why it's removed.

samkerner · 2020-07-16T17:37:16Z

test/run_envoy_bazel_coverage.sh

+  COVERAGE_TARGETS=//test/...
 fi

+if [[ "${FUZZ_COVERAGE}" == "true" ]]; then


This if has the same condition as the next if block. Can they be merged?

lizan · 2020-07-16T21:17:47Z

ci/do_ci.sh

+
+  # Reduce the amount of memory Bazel tries to use to prevent it from launching too many subprocesses.
+  # This should prevent the system from running out of memory and killing tasks. See discussion on
+  # https://github.com/envoyproxy/envoy/pull/5611.
+  [ -z "$CIRCLECI" ] || export BAZEL_BUILD_OPTIONS="${BAZEL_BUILD_OPTIONS} --local_ram_resources=12288"
+


.azure-pipelines/pipelines.yml

Signed-off-by: Asra Ali <asraa@google.com>

Commit Message: Add `bazel.fuzz_coverage` to CI. Publishes fuzz coverage report in `coverage_publish` Risk Level: Medium Testing: Local testing of `./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.fuzz_coverage'`, as well as `FUZZ_COVERAGE=true test/run_envoy_bazel_coverage.sh` with single test args/directory, VALIDATE on/off. Fixes: envoyproxy#9573 Docs: Updated bazel/README.md and fuzz/README.md for docs about running local coverage. Signed-off-by: Asra Ali <asraa@google.com> Signed-off-by: Kevin Baichoo <kbaichoo@google.com>

Commit Message: Add `bazel.fuzz_coverage` to CI. Publishes fuzz coverage report in `coverage_publish` Risk Level: Medium Testing: Local testing of `./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.fuzz_coverage'`, as well as `FUZZ_COVERAGE=true test/run_envoy_bazel_coverage.sh` with single test args/directory, VALIDATE on/off. Fixes: envoyproxy#9573 Docs: Updated bazel/README.md and fuzz/README.md for docs about running local coverage. Signed-off-by: Asra Ali <asraa@google.com> Signed-off-by: scheler <santosh.cheler@appdynamics.com>

add fuzz coverage CI check

6cf628a

Signed-off-by: Asra Ali <asraa@google.com>

add workflow

b1e585f

Signed-off-by: Asra Ali <asraa@google.com>

repokitteh-read-only bot added the waiting label May 5, 2020

asraa added 2 commits May 5, 2020 12:36

Merge remote-tracking branch 'upstream/master' into add-fuzz-ci

1c4a2db

Signed-off-by: Asra Ali <asraa@google.com>

JUST FOR CI FAILURE

c30d149

Signed-off-by: Asra Ali <asraa@google.com>

repokitteh-read-only bot removed the waiting label May 8, 2020

just make this mergeable with gtest

89b95da

Signed-off-by: Asra Ali <asraa@google.com>

asraa changed the title ~~[fuzz] add fuzz coverage CI check~~ [WIP] [fuzz] add fuzz coverage CI check May 11, 2020

stale bot added the stale stalebot believes this issue/PR has not been touched recently label May 19, 2020

Merge remote-tracking branch 'upstream/master' into add-fuzz-ci

ef0701c

Signed-off-by: Asra Ali <asraa@google.com>

stale bot removed the stale stalebot believes this issue/PR has not been touched recently label May 20, 2020

asraa changed the title ~~[WIP] [fuzz] add fuzz coverage CI check~~ [fuzz] add fuzz coverage CI check May 20, 2020

asraa marked this pull request as ready for review May 21, 2020 12:03

fix dependency

f9266bb

Signed-off-by: Asra Ali <asraa@google.com>

asraa added 2 commits May 21, 2020 11:07

fix coverage publish

9e4b175

Signed-off-by: Asra Ali <asraa@google.com>

Merge remote-tracking branch 'upstream/master' into add-fuzz-ci

da63fbb

Signed-off-by: Asra Ali <asraa@google.com>

asraa mentioned this pull request May 21, 2020

Support fuzzing encoder callbacks in UberFilterFuzzer #11209

Merged

asraa added 3 commits May 21, 2020 14:28

fix

2ce6603

Signed-off-by: Asra Ali <asraa@google.com>

fixup

baa334f

Signed-off-by: Asra Ali <asraa@google.com>

hopefully all fixed

90f49ab

Signed-off-by: Asra Ali <asraa@google.com>

ready for review

11850bf

Signed-off-by: Asra Ali <asraa@google.com>

asraa added 5 commits June 25, 2020 11:16

max size and leak detect

25bac35

Signed-off-by: Asra Ali <asraa@google.com>

remove libc++ workaround

ad35385

Signed-off-by: Asra Ali <asraa@google.com>

Merge remote-tracking branch 'upstream/master' into add-fuzz-ci

aebe031

Signed-off-by: Asra Ali <asraa@google.com>

test bin

3b2dddd

Signed-off-by: Asra Ali <asraa@google.com>

add limits and fix script

4d3598c

Signed-off-by: Asra Ali <asraa@google.com>

asraa closed this Jul 1, 2020

Merge remote-tracking branch 'upstream/master' into add-fuzz-ci

cb9b593

Signed-off-by: Asra Ali <asraa@google.com>

asraa reopened this Jul 13, 2020

asraa added 5 commits July 13, 2020 10:38

fix fuzz filtering

59ce7c4

Signed-off-by: Asra Ali <asraa@google.com>

fix script

a33f522

Signed-off-by: Asra Ali <asraa@google.com>

no more exception issues

d77c18b

Signed-off-by: Asra Ali <asraa@google.com>

Merge remote-tracking branch 'upstream/master' into add-fuzz-ci

bbaaf58

Signed-off-by: Asra Ali <asraa@google.com>

fix router fuzz test

a4e3e2f

Signed-off-by: Asra Ali <asraa@google.com>

sqkerner reviewed Jul 16, 2020

View reviewed changes

test/run_envoy_bazel_coverage.sh Show resolved Hide resolved

asraa added 2 commits July 16, 2020 13:46

address comment

f94c392

Signed-off-by: Asra Ali <asraa@google.com>

fix syntax

7ea6e16

Signed-off-by: Asra Ali <asraa@google.com>

samkerner reviewed Jul 16, 2020

View reviewed changes

lizan reviewed Jul 16, 2020

View reviewed changes

address comments

4a212b2

Signed-off-by: Asra Ali <asraa@google.com>

lizan approved these changes Jul 18, 2020

View reviewed changes

lizan merged commit c077fa1 into envoyproxy:master Jul 19, 2020

sqkerner mentioned this pull request Aug 1, 2020

Building a fuzz test with coverage fails with error relocation R_X86_64_PC32 out of range #11802

Closed

Conversation

asraa commented May 4, 2020

Uh oh!

asraa commented May 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

htuch commented May 5, 2020

Uh oh!

htuch commented May 5, 2020

Uh oh!

asraa commented May 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lizan commented May 8, 2020

Uh oh!

asraa commented May 11, 2020

Uh oh!

stale bot commented May 19, 2020

Uh oh!

asraa commented May 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asraa commented May 21, 2020

Uh oh!

asraa commented May 26, 2020

Uh oh!

asraa commented Jun 25, 2020

Uh oh!

asraa commented Jul 1, 2020

Uh oh!

asraa commented Jul 1, 2020

Uh oh!

Uh oh!

asraa commented Jul 16, 2020

Uh oh!

samkerner Jul 16, 2020

Choose a reason for hiding this comment

Uh oh!

lizan Jul 16, 2020

Choose a reason for hiding this comment

Uh oh!

asraa Jul 17, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

asraa commented May 4, 2020 •

edited

Loading

asraa commented May 8, 2020 •

edited

Loading

asraa commented May 21, 2020 •

edited

Loading