http: fix http/1 client low watermark crash

[mattklein123]

This is a regression from #10561.

Fixes #10655

Risk Level: Low (for this fix, the original change was/is still medium)
Testing: New UT
Docs Changes: N/A
Release Notes: N/A

[mattklein123]

cc @euroelessar @rgs1

[rgs1]

@mattklein123 thanks! I'll give it a spin with real traffic tomorrow (but if you want to merge before, that's good too).

[mattklein123]

@htuch @asraa I've re-run TSAN 3 times and it keeps failing on:

[2020-04-06 02:23:47.923][156][critical][assert] [source/common/event/file_event_impl.cc:65] assert failure: events.
[2020-04-06 02:23:47.923][156][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0xfffe00000011
[2020-04-06 02:23:47.923][156][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-04-06 02:23:47.923][156][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 0/1.14.0-dev/redacted/DEBUG/BoringSSL
[2020-04-06 02:23:47.943][150][info][main] [source/server/server.cc:613] shutting down server instance
[2020-04-06 02:23:47.946][150][info][main] [source/server/server.cc:560] main dispatch loop exited
[2020-04-06 02:23:48.031][156][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x5a8e817]
[2020-04-06 02:23:48.132][156][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: __tsan::CallUserSignalHandler() [0x2684350]
external/bazel_tools/tools/test/test-setup.sh: line 310:    17 Aborted                 (core dumped) "${TEST_PATH}" "$@" 2>&1
================================================================================
FAIL: //test/integration:h1_capture_fuzz_test (see /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-dbg/testlogs/test/integration/h1_capture_fuzz_test/test.log)
INFO: From Testing //test/integration:h1_capture_fuzz_test:

I tried running this locally in TSAN mode and it passed. I'm at a loss for how my change would effect this test in this way either way. Any ideas here?

[mattklein123]

Nevermind I did get this to reproduce locally with a non-RBE build. Looking.

[mattklein123]

Hmm, I think I'm hitting libevent/libevent#984 and #8199, but I don't understand how my patch would effect this. Also, it seems to fail sporadically so maybe tickling a timing issue. @htuch any suggestions?

[alyssawilk]

You could try flake instructions before and after and see if it's statistically worse?

[mattklein123]

@alyssawilk updated w/ merged flake fix.

[alyssawilk]

Thanks for tackling this and thanks even more for the "we need to clean this up" comment because we really need to clean that up :-/