Skip to content

upstream: fix panic on grpc unknown_service status on healthchecks #10863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattklein123 merged 4 commits into from
Apr 21, 2020
Merged

upstream: fix panic on grpc unknown_service status on healthchecks #10863

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a54f616
upstream: fix panic on grpc unknown_service status on healthchecks
Sh4d1 Apr 20, 2020
e96178e
add failsafe for non healtcheck enums
Sh4d1 Apr 21, 2020
fc1fd35
fix review
Sh4d1 Apr 21, 2020
889d212
fix format
Sh4d1 Apr 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/root/version_history/current.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Changes
Can be reverted temporarily by setting runtime feature `envoy.reloadable_features.fix_upgrade_response` to false.
* tracing: tracing configuration has been made fully dynamic and every HTTP connection manager
can now have a separate :ref:`tracing provider <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.Tracing.provider>`.
* upstream: fixed a bug where Envoy would panic when receiving a GRPC SERVICE_UNKNOWN status on the health check.

Deprecated
----------
Expand Down
6 changes: 4 additions & 2 deletions source/common/upstream/health_checker_impl.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -793,9 +793,11 @@ void GrpcHealthCheckerImpl::GrpcActiveHealthCheckSession::logHealthCheckStatus(
case grpc::health::v1::HealthCheckResponse::UNKNOWN:
service_status = "unknown";
break;
case grpc::health::v1::HealthCheckResponse::SERVICE_UNKNOWN:
service_status = "service_unknown";
break;
default:
// Should not happen really, Protobuf should not parse undefined enums values.
NOT_REACHED_GCOVR_EXCL_LINE;
service_status = "unknown_healthcheck_response";
break;
}
}
Expand Down
28 changes: 28 additions & 0 deletions test/common/upstream/health_checker_impl_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4452,6 +4452,34 @@ TEST_F(GrpcHealthCheckerImplTest, GrpcFailUnknown) {
cluster_->prioritySet().getMockHostSet(0)->hosts_[0]->health());
}

// Test SERVICE_UNKNOWN health status is considered unhealthy.
TEST_F(GrpcHealthCheckerImplTest, GrpcFailServiceUnknown) {
setupHC();
expectSingleHealthcheck(HealthTransition::Changed);
EXPECT_CALL(*event_logger_, logEjectUnhealthy(_, _, _));
EXPECT_CALL(*event_logger_, logUnhealthy(_, _, _, true));

respondServiceStatus(0, grpc::health::v1::HealthCheckResponse::SERVICE_UNKNOWN);
EXPECT_TRUE(cluster_->prioritySet().getMockHostSet(0)->hosts_[0]->healthFlagGet(
Host::HealthFlag::FAILED_ACTIVE_HC));
EXPECT_EQ(Host::Health::Unhealthy,
cluster_->prioritySet().getMockHostSet(0)->hosts_[0]->health());
}

// Test non existent health status enum is considered unhealthy.
TEST_F(GrpcHealthCheckerImplTest, GrpcFailUnknownHealthStatus) {
setupHC();
expectSingleHealthcheck(HealthTransition::Changed);
EXPECT_CALL(*event_logger_, logEjectUnhealthy(_, _, _));
EXPECT_CALL(*event_logger_, logUnhealthy(_, _, _, true));

respondServiceStatus(0, static_cast<grpc::health::v1::HealthCheckResponse::ServingStatus>(999));
EXPECT_TRUE(cluster_->prioritySet().getMockHostSet(0)->hosts_[0]->healthFlagGet(
Host::HealthFlag::FAILED_ACTIVE_HC));
EXPECT_EQ(Host::Health::Unhealthy,
cluster_->prioritySet().getMockHostSet(0)->hosts_[0]->health());
}

// Test receiving GOAWAY is interpreted as connection close event.
TEST_F(GrpcHealthCheckerImplTest, GoAwayProbeInProgress) {
// FailureType::Network will be issued, it will render host unhealthy only if unhealthy_threshold
Expand Down