Skip to content

test: run fake upstream initialization from the dispatcher thread#22856

Merged
mattklein123 merged 1 commit intoenvoyproxy:mainfrom
benjaminp:fake-upstream-race
Aug 29, 2022
Merged

test: run fake upstream initialization from the dispatcher thread#22856
mattklein123 merged 1 commit intoenvoyproxy:mainfrom
benjaminp:fake-upstream-race

Conversation

@benjaminp
Copy link
Contributor

//test/integration:tcp_proxy_odcds_integration_test was observed to fail as follows:

==================== Test output for //test/integration:tcp_proxy_odcds_integration_test:
[==========] Running 24 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 24 tests from IpVersionsClientType/TcpProxyOdcdsIntegrationTest
[ RUN      ] IpVersionsClientType/TcpProxyOdcdsIntegrationTest.SingleTcpClient/0
[2022-08-25 20:22:46.750][3969][critical][assert] [test/integration/fake_upstream.cc:832] assert failure: !dispatcher_->isThreadSafe().
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6b00000f81
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x4c46a78]->[0x2cf2a78] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1267
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7ffbdaa79420]->[0x7ffbd8b25420] ??:0
[2022-08-25 20:22:46.802][3969][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::FakeUpstream::assertPendingConnectionsEmpty() [0x245bf0b]->[0x507f0b] ??:0
[2022-08-25 20:22:46.846][3969][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::(anonymous namespace)::TcpProxyOdcdsIntegrationTest_SingleTcpClient_Test::TestBody() [0x1f596cb]->[0x56cb] ??:0
[2022-08-25 20:22:46.877][3969][critical][backtrace] [./source/server/backtrace.h:96] #4: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588b61b]->[0x393761b] ??:0
[2022-08-25 20:22:46.924][3969][critical][backtrace] [./source/server/backtrace.h:96] #5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587c2bd]->[0x39282bd] ??:0
[2022-08-25 20:22:46.966][3969][critical][backtrace] [./source/server/backtrace.h:96] #6: testing::Test::Run() [0x5864ba3]->[0x3910ba3] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:0
[2022-08-25 20:22:47.003][3969][critical][backtrace] [./source/server/backtrace.h:96] #7: testing::TestInfo::Run() [0x586576a]->[0x391176a] external/com_google_absl/absl/container/internal/raw_hash_set.h:1259
[2022-08-25 20:22:47.037][3969][critical][backtrace] [./source/server/backtrace.h:96] #8: testing::TestSuite::Run() [0x5865fbb]->[0x3911fbb] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_algo.h:1925
[2022-08-25 20:22:47.089][3969][critical][backtrace] [./source/server/backtrace.h:96] #9: testing::internal::UnitTestImpl::RunAllTests() [0x5874a28]->[0x3920a28] envoy/registry/registry.h:509
[2022-08-25 20:22:47.114][3969][critical][backtrace] [./source/server/backtrace.h:96] #10: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588ddbb]->[0x3939dbb] envoy/registry/registry.h:0
[2022-08-25 20:22:47.160][3969][critical][backtrace] [./source/server/backtrace.h:96] #11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587e683]->[0x392a683] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154
[2022-08-25 20:22:47.191][3969][critical][backtrace] [./source/server/backtrace.h:96] #12: testing::UnitTest::Run() [0x5874568]->[0x3920568] envoy/registry/registry.h:508
[2022-08-25 20:22:47.237][3969][critical][backtrace] [./source/server/backtrace.h:96] #13: RUN_ALL_TESTS() [0x4878d51]->[0x2924d51] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1203
[2022-08-25 20:22:47.284][3969][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::TestRunner::RunTests() [0x48783b1]->[0x29243b1] external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:485
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #15: main [0x4874c3a]->[0x2920c3a] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1181
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #16: __libc_start_main [0x7ffbda897083]->[0x7ffbd8943083] ??:0
================================================================================

This is due to the race described by #22855. Making sure the dispatcher thread is running before starting the test avoids this problem.

Signed-off-by: Benjamin Peterson benjamin@engflow.com

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

`//test/integration:tcp_proxy_odcds_integration_test` was observed to fail as follows:
```
==================== Test output for //test/integration:tcp_proxy_odcds_integration_test:
[==========] Running 24 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 24 tests from IpVersionsClientType/TcpProxyOdcdsIntegrationTest
[ RUN      ] IpVersionsClientType/TcpProxyOdcdsIntegrationTest.SingleTcpClient/0
[2022-08-25 20:22:46.750][3969][critical][assert] [test/integration/fake_upstream.cc:832] assert failure: !dispatcher_->isThreadSafe().
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6b00000f81
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x4c46a78]->[0x2cf2a78] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1267
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#1: __restore_rt [0x7ffbdaa79420]->[0x7ffbd8b25420] ??:0
[2022-08-25 20:22:46.802][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#2: Envoy::FakeUpstream::assertPendingConnectionsEmpty() [0x245bf0b]->[0x507f0b] ??:0
[2022-08-25 20:22:46.846][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#3: Envoy::(anonymous namespace)::TcpProxyOdcdsIntegrationTest_SingleTcpClient_Test::TestBody() [0x1f596cb]->[0x56cb] ??:0
[2022-08-25 20:22:46.877][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#4: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588b61b]->[0x393761b] ??:0
[2022-08-25 20:22:46.924][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587c2bd]->[0x39282bd] ??:0
[2022-08-25 20:22:46.966][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#6: testing::Test::Run() [0x5864ba3]->[0x3910ba3] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:0
[2022-08-25 20:22:47.003][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#7: testing::TestInfo::Run() [0x586576a]->[0x391176a] external/com_google_absl/absl/container/internal/raw_hash_set.h:1259
[2022-08-25 20:22:47.037][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#8: testing::TestSuite::Run() [0x5865fbb]->[0x3911fbb] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_algo.h:1925
[2022-08-25 20:22:47.089][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#9: testing::internal::UnitTestImpl::RunAllTests() [0x5874a28]->[0x3920a28] envoy/registry/registry.h:509
[2022-08-25 20:22:47.114][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#10: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588ddbb]->[0x3939dbb] envoy/registry/registry.h:0
[2022-08-25 20:22:47.160][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587e683]->[0x392a683] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154
[2022-08-25 20:22:47.191][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#12: testing::UnitTest::Run() [0x5874568]->[0x3920568] envoy/registry/registry.h:508
[2022-08-25 20:22:47.237][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#13: RUN_ALL_TESTS() [0x4878d51]->[0x2924d51] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1203
[2022-08-25 20:22:47.284][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#14: Envoy::TestRunner::RunTests() [0x48783b1]->[0x29243b1] external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:485
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#15: main [0x4874c3a]->[0x2920c3a] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1181
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#16: __libc_start_main [0x7ffbda897083]->[0x7ffbd8943083] ??:0
================================================================================
```

This is due to the race described by envoyproxy#22855. Making sure the dispatcher thread is running before starting the test avoids this problem.

Signed-off-by: Benjamin Peterson <benjamin@engflow.com>
@mattklein123 mattklein123 self-assigned this Aug 29, 2022
Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thank you.

@mattklein123
Copy link
Member

/retest

@repokitteh-read-only
Copy link

Retrying Azure Pipelines:
Check envoy-presubmit didn't fail.

🐱

Caused by: a #22856 (comment) was created by @mattklein123.

see: more, trace.

@mattklein123 mattklein123 merged commit 97af8e5 into envoyproxy:main Aug 29, 2022
@benjaminp benjaminp deleted the fake-upstream-race branch August 29, 2022 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants