Skip to content

core: fix shutdown libevent crash#527

Merged
junr03 merged 3 commits intomasterfrom
crash-fix
Oct 21, 2019
Merged

core: fix shutdown libevent crash#527
junr03 merged 3 commits intomasterfrom
crash-fix

Conversation

@junr03
Copy link
Member

@junr03 junr03 commented Oct 19, 2019 •
edited
Loading

Description: #498 did not fully solve #492. The reset cleanly destructed all objects. However, because destruction was posted in to the event dispatcher, the event dispatcher was left with bad accesses.

This PR fixes the issue by issuing shutdown on the dispatcher, and only destructing once the event loop has exited and control has returned to the Engine's run function.

Risk Level: med - fixing crash on shutdown
Testing: local

Fixes #492

Signed-off-by: Jose Nino jnino@lyft.com

Jose Nino added 3 commits October 18, 2019 17:33
core: fix shutdown libevent crash
988e7e7 
Signed-off-by: Jose Nino <jnino@lyft.com>
result
872ab49 
Signed-off-by: Jose Nino <jnino@lyft.com>
Merge branch 'master' into crash-fix
30207ab 
Signed-off-by: Jose Nino <jnino@lyft.com>
@rebello95
Copy link
Contributor

rebello95 commented Oct 21, 2019

Tested and was able to see the following graceful shutdown:

2019-10-21 09:54:58.212548-0700 Lyft[86755:6119208] Envoy is exiting.
[2019-10-21 09:54:58.212][6119208][info][main] [external/envoy/source/server/server.cc:602] shutting down server instance
[2019-10-21 09:54:58.213][6119208][info][main] [external/envoy/source/server/server.cc:556] main dispatch loop exited
[2019-10-21 09:54:58.213][6119266][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.214][6119266][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.215][6119267][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.215][6119267][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.215][6119268][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.215][6119268][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.231][6119269][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.231][6119269][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.231][6119270][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.231][6119270][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.231][6119271][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.231][6119271][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119272][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.232][6119272][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119273][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.232][6119273][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119274][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.232][6119274][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119275][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.232][6119275][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119276][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.232][6119276][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119277][debug][main] [external/envoy/source/server/worker_impl.cc:111] worker exited dispatch loop
[2019-10-21 09:54:58.232][6119277][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.232][6119208][debug][main] [external/envoy/source/server/server.cc:173] flushing stats
[2019-10-21 09:54:58.235][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] ClusterImplBase destroyed
[2019-10-21 09:54:58.235][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Cluster base destroyed
[2019-10-21 09:54:58.235][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] ClusterImplBase destroyed
[2019-10-21 09:54:58.235][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Cluster base_wwan destroyed
[2019-10-21 09:54:58.235][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] ClusterImplBase destroyed
[2019-10-21 09:54:58.235][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Cluster base_wlan destroyed
[2019-10-21 09:54:58.235][6119208][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:940] shutting down thread local cluster manager
[2019-10-21 09:54:58.235][6119208][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C0] closing data_to_write=0 type=1
[2019-10-21 09:54:58.235][6119208][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C0] closing socket: 1
[2019-10-21 09:54:58.236][6119208][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:298] [C0] SSL shutdown: rc=0
[2019-10-21 09:54:58.236][6119208][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C0] 
[2019-10-21 09:54:58.236][6119208][debug][client] [external/envoy/source/common/http/codec_client.cc:83] [C0] disconnect. resetting 2 pending requests
[2019-10-21 09:54:58.236][6119208][debug][client] [external/envoy/source/common/http/codec_client.cc:106] [C0] request reset
[2019-10-21 09:54:58.236][6119208][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=1)
[2019-10-21 09:54:58.236][6119208][debug][pool] [external/envoy/source/common/http/http2/conn_pool.cc:234] [C0] destroying stream: 1 remaining
[2019-10-21 09:54:58.236][6119208][debug][router] [external/envoy/source/common/router/router.cc:877] [C0][S5526673004010548937] upstream reset: reset reason connection termination
[2019-10-21 09:54:58.236][6119208][debug][http] [library/common/http/dispatcher.cc:95] [S39] remote reset stream
[2019-10-21 09:54:58.237][6119208][debug][http] [library/common/http/dispatcher.cc:296] [S39] remove stream
[2019-10-21 09:54:58.237][6119208][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=2)
[2019-10-21 09:54:58.237][6119208][debug][client] [external/envoy/source/common/http/codec_client.cc:106] [C0] request reset
[2019-10-21 09:54:58.237][6119208][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=3)
[2019-10-21 09:54:58.237][6119208][debug][pool] [external/envoy/source/common/http/http2/conn_pool.cc:234] [C0] destroying stream: 0 remaining
[2019-10-21 09:54:58.237][6119208][debug][router] [external/envoy/source/common/router/router.cc:877] [C0][S10464915947485995508] upstream reset: reset reason connection termination
[2019-10-21 09:54:58.237][6119208][debug][http] [library/common/http/dispatcher.cc:95] [S17] remote reset stream
[2019-10-21 09:54:58.237][6119208][debug][http] [library/common/http/dispatcher.cc:296] [S17] remove stream
[2019-10-21 09:54:58.237][6119208][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=4)
[2019-10-21 09:54:58.237][6119208][debug][pool] [external/envoy/source/common/http/http2/conn_pool.cc:150] [C0] client disconnected
[2019-10-21 09:54:58.237][6119208][debug][pool] [external/envoy/source/common/http/http2/conn_pool.cc:172] [C0] destroying primary client
[2019-10-21 09:54:58.237][6119208][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=5)
[2019-10-21 09:54:58.237][6119208][trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:76] clearing deferred deletion list (size=5)
[2019-10-21 09:54:58.237][6119208][trace][http2] [external/envoy/source/common/http/http2/nghttp2.cc:20] nghttp2: send: reset nghttp2_active_outbound_item
[2019-10-21 09:54:58.237][6119208][trace][http2] [external/envoy/source/common/http/http2/nghttp2.cc:20] nghttp2: send: aob->item = 0x0
[2019-10-21 09:54:58.243][6119208][info][main] [external/envoy/source/server/server.cc:595] exiting
[2019-10-21 09:54:58.243][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] RunHelper destroyed
[2019-10-21 09:54:58.248][6119208][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Server destroyed

rebello95
rebello95 approved these changes Oct 21, 2019
View reviewed changes
Copy link
Contributor

@rebello95 rebello95 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's link the original issue in the PR and resolve it as part of this PR as well. Thanks for this!

@junr03 junr03 merged commit 3beee64 into master Oct 21, 2019
@junr03 junr03 deleted the crash-fix branch October 21, 2019 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@rebello95 rebello95 rebello95 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

core/ios: crash when force-closing app with Envoy

2 participants

@junr03 @rebello95