-
Notifications
You must be signed in to change notification settings - Fork 5.5k
health check: gracefully handle GOAWAY in grpc health checker #11324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 13 commits
e4ddb8e
70263a6
a069698
725caf8
58d2143
00a1fea
1828a2b
64c6aef
6ba45a4
c5958dc
b474edd
9171180
73e7308
dc5effd
7c7d930
41ea1e1
a66c176
bf9c59c
878bd4a
541eb54
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -471,7 +471,7 @@ ConnectionImpl::ConnectionImpl(Network::Connection& connection, CodecStats& stat | |
| http2_options.max_inbound_priority_frames_per_stream().value()), | ||
| max_inbound_window_update_frames_per_data_frame_sent_( | ||
| http2_options.max_inbound_window_update_frames_per_data_frame_sent().value()), | ||
| dispatching_(false), raised_goaway_(false), pending_deferred_reset_(false) {} | ||
| dispatching_(false), pending_deferred_reset_(false) {} | ||
|
|
||
| ConnectionImpl::~ConnectionImpl() { nghttp2_session_del(session_); } | ||
|
|
||
|
|
@@ -563,6 +563,16 @@ int ConnectionImpl::onBeforeFrameReceived(const nghttp2_frame_hd* hd) { | |
| return 0; | ||
| } | ||
|
|
||
| ABSL_MUST_USE_RESULT | ||
| enum ErrorCode ngHttp2ErrorCodeToErrorCode(uint32_t code) noexcept { | ||
| switch (code) { | ||
| case NGHTTP2_NO_ERROR: | ||
| return ErrorCode::NoError; | ||
| default: | ||
| return ErrorCode::Other; | ||
| } | ||
| } | ||
|
|
||
| int ConnectionImpl::onFrameReceived(const nghttp2_frame* frame) { | ||
| ENVOY_CONN_LOG(trace, "recv frame type={}", connection_, static_cast<uint64_t>(frame->hd.type)); | ||
|
|
||
|
|
@@ -577,12 +587,10 @@ int ConnectionImpl::onFrameReceived(const nghttp2_frame* frame) { | |
| } | ||
| } | ||
|
|
||
| // Only raise GOAWAY once, since we don't currently expose stream information. Shutdown | ||
| // notifications are the same as a normal GOAWAY. | ||
| if (frame->hd.type == NGHTTP2_GOAWAY && !raised_goaway_) { | ||
| // Shutdown notifications are the same as a normal GOAWAY. | ||
| if (frame->hd.type == NGHTTP2_GOAWAY) { | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure if this is fine or if we want to track which goaway error codes we've already seen, and run the callbacks once per error code.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it expected to get multiple GOAWAY frames with different error codes? If yes, it seems bad only look at the first one, if no then we can probably keep the old logic.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is possible to receive GOAWAY frames with different error codes:
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm in that case should the logic for failing the HC be that we see a NO_ERROR + no error GOAWAYs? Also I'm not sure what the implications are of making
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Hm, I don't quite follow, could you rephrase?
Agreed. It seems it was added in #103. I'm not sure if that was an optimization back then. I'm also not sure if it has become an expected behaviour (intentionally or not) since then.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if a GOAWAY error follows a NO_ERROR then
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, I see. I don't expect that to change the behaviour as it stands:
I admit this isn't obvious / does feel a bit brittle -- happy to add a comment and/or reset the flag anyways.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Friendly ping @mattklein123 @alyssawilk
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry for the delay. My preference would be to not raise multiple goaways from the codec as I'm guessing there will be code that won't expect this and will break. Can we just keep the previous behavior and not handle the case where multiple go away frames are received? We can potentially deal with this in a follow up if someone requests it?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the feedback - that seems reasonable to me. I've reverted that specific commit and added a TODO instead. |
||
| ASSERT(frame->hd.stream_id == 0); | ||
| raised_goaway_ = true; | ||
| callbacks().onGoAway(); | ||
| callbacks().onGoAway(ngHttp2ErrorCodeToErrorCode(frame->goaway.error_code)); | ||
| return 0; | ||
| } | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -703,19 +703,22 @@ void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onInterval() { | |
| void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onResetStream(Http::StreamResetReason, | ||
| absl::string_view) { | ||
| const bool expected_reset = expect_reset_; | ||
| const bool goaway = received_goaway_; | ||
| resetState(); | ||
|
|
||
| if (expected_reset) { | ||
| // Stream reset was initiated by us (bogus gRPC response, timeout or cluster host is going | ||
| // away). In these cases health check failure has already been reported, so just return. | ||
| // away). In these cases health check failure has already been reported and a GOAWAY (if any) | ||
| // has already been handled, so just return. | ||
| return; | ||
| } | ||
|
|
||
| ENVOY_CONN_LOG(debug, "connection/stream error health_flags={}", *client_, | ||
| HostUtility::healthFlagsToString(*host_)); | ||
|
|
||
| if (!parent_.reuse_connection_) { | ||
| // Stream reset was unexpected, so we haven't closed the connection yet. | ||
| if (goaway || !parent_.reuse_connection_) { | ||
| // Stream reset was unexpected, so we haven't closed the connection | ||
| // yet in response to a GOAWAY or due to disabled connection reuse. | ||
| client_->close(); | ||
| } | ||
|
|
||
|
|
@@ -727,9 +730,18 @@ void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onResetStream(Http::St | |
| handleFailure(envoy::data::core::v3::NETWORK); | ||
| } | ||
|
|
||
| void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onGoAway() { | ||
| void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onGoAway(Http::ErrorCode error_code) { | ||
| ENVOY_CONN_LOG(debug, "connection going away health_flags={}", *client_, | ||
| HostUtility::healthFlagsToString(*host_)); | ||
| // If we have an active health check probe and receive a GOAWAY indicating | ||
| // graceful shutdown, allow the probe to complete before closing the connection. | ||
| // The connection will be closed when the active check completes or another | ||
| // terminal condition occurs, such as a timeout or stream reset. | ||
| if (request_encoder_ && error_code == Http::ErrorCode::NoError) { | ||
| received_goaway_ = true; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A one liner comment here about when this connection would eventually get closed would be nice. IIUC it's either on the next timeout or when the remote closes it?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll add a comment. It would be closed when the active check completes (
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe rename this to make it clear that we received a non-error GOAWAY? |
||
| return; | ||
| } | ||
|
|
||
| // Even if we have active health check probe, fail it on GOAWAY and schedule new one. | ||
| if (request_encoder_) { | ||
| handleFailure(envoy::data::core::v3::NETWORK); | ||
|
|
@@ -762,6 +774,9 @@ void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onRpcComplete( | |
| handleFailure(envoy::data::core::v3::ACTIVE); | ||
| } | ||
|
|
||
| // Read the value as we may call resetState() and clear it. | ||
| const bool goaway = received_goaway_; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mind adding a comment here explaining why we have to read the value here? Its a bit less clear than on L705 since we're not unconditionally calling |
||
|
|
||
| // |end_stream| will be false if we decided to stop healthcheck before HTTP stream has ended - | ||
| // invalid gRPC payload, unexpected message stream or wrong content-type. | ||
| if (end_stream) { | ||
|
|
@@ -772,7 +787,7 @@ void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onRpcComplete( | |
| request_encoder_->getStream().resetStream(Http::StreamResetReason::LocalReset); | ||
| } | ||
|
|
||
| if (!parent_.reuse_connection_) { | ||
| if (!parent_.reuse_connection_ || goaway) { | ||
| client_->close(); | ||
| } | ||
| } | ||
|
|
@@ -782,13 +797,14 @@ void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::resetState() { | |
| request_encoder_ = nullptr; | ||
| decoder_ = Grpc::Decoder(); | ||
| health_check_response_.reset(); | ||
| received_goaway_ = false; | ||
| } | ||
|
|
||
| void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::onTimeout() { | ||
| ENVOY_CONN_LOG(debug, "connection/stream timeout health_flags={}", *client_, | ||
| HostUtility::healthFlagsToString(*host_)); | ||
| expect_reset_ = true; | ||
|
snowp marked this conversation as resolved.
|
||
| if (!parent_.reuse_connection_) { | ||
| if (received_goaway_ || !parent_.reuse_connection_) { | ||
| client_->close(); | ||
| } else { | ||
| request_encoder_->getStream().resetStream(Http::StreamResetReason::LocalReset); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -317,7 +317,8 @@ TEST_P(Http2CodecImplTest, ShutdownNotice) { | |
| EXPECT_CALL(request_decoder_, decodeHeaders_(_, true)); | ||
| request_encoder_->encodeHeaders(request_headers, true); | ||
|
|
||
| EXPECT_CALL(client_callbacks_, onGoAway()); | ||
| // Called once from the shutdown notice and once from the goaway. | ||
| EXPECT_CALL(client_callbacks_, onGoAway(_)).Times(2); | ||
| server_->shutdownNotice(); | ||
| server_->goAway(); | ||
|
|
||
|
|
@@ -1456,7 +1457,7 @@ TEST_P(Http2CodecImplTest, LargeRequestHeadersExceedPerHeaderLimit) { | |
| request_headers.addCopy("big", long_string); | ||
|
|
||
| EXPECT_CALL(request_decoder_, decodeHeaders_(_, _)).Times(0); | ||
| EXPECT_CALL(client_callbacks_, onGoAway()); | ||
| EXPECT_CALL(client_callbacks_, onGoAway(_)); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't completely follow why this test doesn't trigger |
||
| server_->shutdownNotice(); | ||
| server_->goAway(); | ||
| request_encoder_->encodeHeaders(request_headers, true); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe call this
GoAwayErrorCode? to make it clearer what kind of error codes are being modeled hereThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 will update. Initially this enumerated all the error codes here, which are shared with RST_STREAM. But we've since undone that and there's already a separate Envoy-internal enum for stream resets.