Skip to content

thrift proxy: add stats to the router filter#9203

Merged
zuercher merged 2 commits intoenvoyproxy:masterfrom
rgs1:issue-9115
Dec 4, 2019
Merged

thrift proxy: add stats to the router filter#9203
zuercher merged 2 commits intoenvoyproxy:masterfrom
rgs1:issue-9115

Conversation

@rgs1
Copy link
Member

@rgs1 rgs1 commented Dec 3, 2019

This adds stats for missing route, missing cluster, no healthy hosts,
etc. Should be useful to protect operators against bad config pushes,
plus gives more visibility.

Signed-off-by: Raul Gutierrez Segales rgs@pinterest.com

This adds stats for missing route, missing cluster, no healthy hosts,
etc. Should be useful to protect operators against bad config pushes,
plus gives more visibility.

Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
@rgs1
Copy link
Member Author

rgs1 commented Dec 3, 2019

failure seems unrelated:

//test/server:server_fuzz_test                                           FAILED in 81.3s
  /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/execroot/envoy/bazel-out/k8-dbg/testlogs/test/server/server_fuzz_test/test.log

Test log:

[2019-12-03 17:57:00.785][17][info][upstream] [source/common/upstream/cluster_manager_impl.cc:167] cm init: all clusters initialized
/opt/llvm/bin/../include/c++/v1/chrono:888:59: runtime error: signed integer overflow: 12135157018181088 * 1000 cannot be represented in type 'long long'
    #0 0x15863ea8 in std::__1::chrono::__duration_cast<std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, std::__1::chrono::duration<long long, std::__1::ratio<1l,
 1000000l> >, std::__1::ratio<1000l, 1l>, false, true>::operator()(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&) const /opt/llvm/bin/../include/c++/v1/
chrono:888:59
    #1 0x15863c12 in std::__1::enable_if<__is_duration<std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l> > >::value, std::__1::chrono::duration<long long, std::__
1::ratio<1l, 1000000l> > >::type std::__1::chrono::duration_cast<std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l> >, long long, std::__1::ratio<1l, 1000l> >(std:
:__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&) /opt/llvm/bin/../include/c++/v1/chrono:915:12
    #2 0x158639dc in std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l> >::duration<long long, std::__1::ratio<1l, 1000l> >(std::__1::chrono::duration<long long, s
td::__1::ratio<1l, 1000l> > const&, std::__1::enable_if<(__no_overflow<std::__1::ratio<1l, 1000l>, std::__1::ratio<1l, 1000000l> >::value) && ((std::__1::integral_constant<bool, fa
lse>::value) || (((__no_overflow<std::__1::ratio<1l, 1000l>, std::__1::ratio<1l, 1000000l> >::type::den) == (1)) && (!(treat_as_floating_point<long long>::value)))), void>::type*) 
/opt/llvm/bin/../include/c++/v1/chrono:1063:26
    #3 0x22f3d22e in Envoy::Event::TimerImpl::enableTimer(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, Envoy::ScopeTrackedObject const*) /proc/self/cw
d/source/common/event/timer_impl.cc:41:17
    #4 0x1daa5e3e in Envoy::Upstream::HealthCheckerImplBase::ActiveHealthCheckSession::handleFailure(envoy::data::core::v2alpha::HealthCheckFailureType) /proc/self/cwd/source/commo
n/upstream/health_checker_base_impl.cc:351:22
    #5 0x1da37146 in Envoy::Upstream::TcpHealthCheckerImpl::TcpActiveHealthCheckSession::onEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/source/common/upstream/health_check
er_impl.cc:441:7
    #6 0x1da5f6d8 in Envoy::Upstream::TcpHealthCheckerImpl::TcpSessionCallbacks::onEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/bazel-out/k8-dbg/bin/source/common/upstream
/_virtual_includes/health_checker_lib/common/upstream/health_checker_impl.h:228:69
    #7 0x21cb0b07 in Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/source/common/network/connection_impl_base.cc:37:15
    #8 0x21c67736 in Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/source/common/network/connection_impl.cc:332:23
....

Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
@rgs1
Copy link
Member Author

rgs1 commented Dec 3, 2019

cc: @oschaaf for the above failure

Copy link
Member

@zuercher zuercher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I have thing to think about, but otherwise looks good.

@zuercher zuercher merged commit d8e234e into envoyproxy:master Dec 4, 2019
rgs1 pushed a commit to rgs1/envoy that referenced this pull request Jan 31, 2020
We've been using this in prod for a couple of months and for
a few different services.

A few critical bugs have been fixed (envoyproxy#9089 and envoyproxy#6549) and some
necessary stats/features have been added (envoyproxy#9203 and envoyproxy#8994), so
it's probably a good time to graduate this filter.

Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
rgs1 pushed a commit to rgs1/envoy that referenced this pull request Feb 1, 2020
We've been using this in prod for a couple of months and for
a few different services.

A few critical bugs have been fixed (envoyproxy#9089 and envoyproxy#6549) and some
necessary stats/features have been added (envoyproxy#9203 and envoyproxy#8994), so
it's probably a good time to graduate at least the proxy
and the router filter.

Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
mattklein123 pushed a commit that referenced this pull request Feb 1, 2020
We've been using this in prod for a couple of months and for
a few different services.

A few critical bugs have been fixed (#9089 and #6549) and some
necessary stats/features have been added (#9203 and #8994), so
it's probably a good time to graduate at least the proxy
and the router filter.

Signed-off-by: Raul Gutierrez Segales <rgs@pinterest.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants