Skip to content

upstream: log event for outlier detection ejection and connection draining #17879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
snowp merged 5 commits into from
Aug 27, 2021
Merged

upstream: log event for outlier detection ejection and connection draining #17879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f481dbe
upstream: log event when outlier detector ejects a host
Aug 26, 2021
e6e20bb
log event when connections are drained
Aug 26, 2021
956317d
Merge remote-tracking branch 'envoy/main' into outlier-logging
Aug 26, 2021
329fb83
include event for drainConnections
Aug 26, 2021
23f13db
add additional event
Aug 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions source/common/conn_pool/conn_pool_base.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,13 +364,17 @@ void ConnPoolImplBase::drainConnectionsImpl() {
// so all remaining entries in ready_clients_ are serving streams. Move them and all entries
// in busy_clients_ to draining.
while (!ready_clients_.empty()) {
ENVOY_LOG_EVENT(debug, "draining_ready_client", "draining active client {} for cluster {}",
ready_clients_.front()->id(), host_->cluster().name());
transitionActiveClientState(*ready_clients_.front(), ActiveClient::State::DRAINING);
}

// Changing busy_clients_ to DRAINING does not move them between lists,
// so use a for-loop since the list is not mutated.
ASSERT(&owningList(ActiveClient::State::DRAINING) == &busy_clients_);
for (auto& busy_client : busy_clients_) {
ENVOY_LOG_EVENT(debug, "draining_busy_client", "draining busy client {} for cluster {}",
busy_client->id(), host_->cluster().name());
transitionActiveClientState(*busy_client, ActiveClient::State::DRAINING);
}
}
Expand Down
9 changes: 9 additions & 0 deletions source/common/upstream/cluster_manager_impl.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,9 @@ ClusterManagerImpl::loadCluster(const envoy::config::cluster::v3::Cluster& clust
if (new_cluster->outlierDetector() != nullptr) {
new_cluster->outlierDetector()->addChangedStateCb([this](HostSharedPtr host) {
if (host->healthFlagGet(Host::HealthFlag::FAILED_OUTLIER_CHECK)) {
ENVOY_LOG_EVENT(debug, "outlier_detection_ejection",
"host {} in cluster {} was ejected by the outlier detector",
host->address(), host->cluster().name());
postThreadLocalHealthFailure(host);
}
});
Expand Down Expand Up @@ -946,6 +949,9 @@ ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::tcpConnPool(
}

void ClusterManagerImpl::drainConnections(const std::string& cluster) {
ENVOY_LOG_EVENT(debug, "drain_connections_call", "drainConnections called for cluster {}",
cluster);

tls_.runOnAllThreads([cluster](OptRef<ThreadLocalClusterManagerImpl> cluster_manager) {
auto cluster_entry = cluster_manager->thread_local_clusters_.find(cluster);
if (cluster_entry != cluster_manager->thread_local_clusters_.end()) {
Expand All @@ -955,6 +961,9 @@ void ClusterManagerImpl::drainConnections(const std::string& cluster) {
}

void ClusterManagerImpl::drainConnections() {
ENVOY_LOG_EVENT(debug, "drain_connections_call_for_all_clusters",
"drainConnections called for all clusters");

tls_.runOnAllThreads([](OptRef<ThreadLocalClusterManagerImpl> cluster_manager) {
for (const auto& cluster_entry : cluster_manager->thread_local_clusters_) {
cluster_entry.second->drainConnPools();
Expand Down