Add new NH statistic class CircllhistStatistic and SinkableCircllhist… by qqustc · Pull Request #391 · envoyproxy/nighthawk

qqustc · 2020-07-01T02:13:14Z

Add new NH statistic class CircllhistStatistic and SinkableCircllhistStatistic.

CircllhistStatistic uses Circllhist under the hood to compute statistics where Circllhist is used in the implementation of Envoy Histograms. Compared to HdrHistogram Circllhist trades precision for fast performance in merge and insertion according to #115.

SinkableCircllhistStatistic wraps the Envoy::Stats::Histogram interface and is used to flush histogram value to downstream Envoy stats Sinks.

Linked Issues: #344

Testing: unit tests

qqustc · 2020-07-01T03:39:27Z

/retest

repokitteh-read-only · 2020-07-01T03:39:31Z

🙀 failed invoking rebuild of ci/circleci: tsan: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: build: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: asan: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: clang_tidy: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: benchmark: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: docker: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: coverage: 403 Forbidden

🐱

Caused by: a #391 (comment) was created by @qqustc.

see: more, trace.

qqustc · 2020-07-01T03:44:23Z

@oschaaf hi Otto, it seems all CI checks failed again due to same error as last time. I will try to trigger it to rerun, meanwhile please go ahead review the change. Thanks!

oschaaf · 2020-07-01T09:25:58Z

/retest

repokitteh-read-only · 2020-07-01T09:26:06Z

🙀 failed invoking rebuild of ci/circleci: tsan: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: build: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: asan: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: clang_tidy: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: benchmark: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: docker: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: coverage: 403 Forbidden
🙀 failed invoking rebuild of ci/circleci: test: 403 Forbidden

🐱

Caused by: a #391 (comment) was created by @oschaaf.

see: more, trace.

oschaaf

This is awesome, and once we merge this it should be trivial to make all the histograms in NH use circllhist as a statistics backend.

I haven't been able to absorb the whole PR at high detail yet, so I'd love to have another pass at it a little bit later; but flushing out my first comments to get started.

source/common/statistic_impl.h

oschaaf · 2020-07-01T17:49:28Z

source/common/statistic_impl.h

+    return count() == 0 ? std::nan("") : hist_approx_stddev(histogram_);
+  }
+  StatisticPtr combine(const Statistic& statistic) const override;
+  uint64_t significantDigits() const override { return 1; }


This seems a low guarantee; I wonder, can we improve? (HdrStatistic manages 4 significant digits in the way it we set it up).

Discussed offline. Confirmed that circllhist only has 2 digit decimal precision as a result of base 10 algorithm.
#115 (comment)

Since this is out of current work scope, decided to leave the discussion of potential impacts for future work when we try to switch to circllhist

Once we move the implementation into the .cc file, we should add a comment explaining the single digit precision.

This comment may have been missed. Even if we end up not moving this to the .cc file, we should add a comment explaining the choice of "1" as the precision.

Thank you mum4k@.
Added a comment here.

oschaaf · 2020-07-01T18:08:24Z

(I have asked for help with CI in the Slack nighthawk-maintainers channel; I'm still clueless as to what is causing the CI trouble here, and me creating mirror PRs and closing them once we merge isn't sustainable)

oschaaf

looks great to me overall, posting my comments after having a look at this in more detail.

source/common/statistic_impl.cc

test/statistic_test.cc

source/common/statistic_impl.h

test/statistic_test.cc

test/test_data/circllhist_proto_json.gold

oschaaf · 2020-07-02T14:55:37Z

I did the magic trick again, creating a PR from my own repo with the same commits to trigger CI. Let's see what happens.

qqustc · 2020-07-03T01:51:22Z

Thanks Otto for the help! Sorry I'm on vacation till the end of this week, I will try to fix the CI check (currently this is too much hassle :( ) and update the PR next week.

…

On Thu, Jul 2, 2020, 10:55 AM Otto van der Schaaf ***@***.***> wrote: I did the magic trick again, creating a PR from my own repo with the same commits to trigger CI. Let's see what happens. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#391 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH6YKY5HDKWDYHFS43FODTRZSNXTANCNFSM4OM6IZXA> .

…Statistic Signed-off-by: Qin Qin <qqin@google.com> Signed-off-by: qqustc <qqustc@gmail.com> Signed-off-by: Qin Qin <qqin@google.com>

Signed-off-by: Qin Qin <qqin@google.com>

Signed-off-by: qqustc <qqustc@gmail.com> Signed-off-by: Qin Qin <qqin@google.com>

Signed-off-by: Qin Qin <qqin@google.com>

Signed-off-by: Qin Qin <qqin@google.com> Signed-off-by: qqustc <qqustc@gmail.com>

Signed-off-by: Qin Qin <qqin@google.com>

mum4k · 2020-07-08T17:32:34Z

test/run_nighthawk_bazel_coverage.sh

 if [ "$VALIDATE_COVERAGE" == "true" ]
 then
-  COVERAGE_THRESHOLD=98.6
+  COVERAGE_THRESHOLD=98.4


Are we forced to lower the threshold? Which lines / behaviors we can't cover in this PR?

https://14282-180498819-gh.circle-artifacts.com/0/coverage/source/common/statistic_impl.h.gcov.html

The lines not covered are std::string tagExtractedName() const override { NOT_IMPLEMENTED_GCOVR_EXCL_LINE; }, where I added test EXPECT_DEATH(stat.tagExtractedName(), ".*"); but it does not count into coverage. So after talked to Otto, we decided to lower the coverage.

mum4k · 2020-07-09T21:06:35Z

test/statistic_test.cc

+  util.loadFromJson(Envoy::Filesystem::fileSystemForTest().fileReadToEnd(
+                        TestEnvironment::runfilesPath("test/test_data/circllhist_proto_json.gold")),
+                    parsed_json_proto, Envoy::ProtobufMessage::getStrictValidationVisitor());
+  // Instead of comparing proto's, we perform a string-based comparison, because that emits a


A more idiomatic approach would be to compare the proto and output a textual diff only if the comparison fails. That way we will be comparing via the binary representations that are the public API. The string are only for humans to read.

Note that EXPECT_EQ takes in an optional stream which can contain a custom error message - this is where we can inject the readable diff:

https://github.com/google/googletest/blob/master/googletest/docs/primer.md#assertions

Changed to compare the proto. PTAL.

mum4k · 2020-07-09T21:10:50Z

source/common/statistic_impl.h

+ */
+class CircllhistStatistic : public StatisticImpl {
+public:
+  CircllhistStatistic() {


Can we place the implementation of the constructor in the .cc file? It helps to keep the header files more readable and focused on the API.

Same comment applies to the destructor and the other methods and class.

Done. Moved most of the implementation to .cc file and only leave some of the 1 line function in .h file. PTAL

mum4k · 2020-07-09T21:12:46Z

source/common/statistic_impl.h

+    return count() == 0 ? std::nan("") : hist_approx_stddev(histogram_);
+  }
+  StatisticPtr combine(const Statistic& statistic) const override;
+  uint64_t significantDigits() const override { return 1; }


Once we move the implementation into the .cc file, we should add a comment explaining the single digit precision.

mum4k · 2020-07-09T21:16:54Z

source/common/statistic_impl.h

+  nighthawk::client::Statistic toProto(SerializationDomain domain) const override;
+
+private:
+  histogram_t* histogram_;


(optional) Using a raw pointer here opens us to memory leaks and all the usual problems of raw pointers.

Would it be possible to move this to a smart pointer (std::unique_ptr) with a custom deleter?

https://www.bfilipek.com/2016/04/custom-deleters-for-c-smart-pointers.html

Thanks Jakub for the reference. It seems the hist_free(histogram_t *hist) is quite complicated, so I will not move it to a smart pointer for now

mum4k · 2020-07-09T21:23:27Z

source/common/statistic_impl.h

+// Implementation of sinkable Nighthawk Statistic with Circllhist Histogram.
+class SinkableCircllhistStatistic : public SinkableStatistic, public CircllhistStatistic {
+public:
+  SinkableCircllhistStatistic(Envoy::Stats::Scope& scope,


Can we document the constructor?

mum4k · 2020-07-09T21:23:58Z

source/common/statistic_impl.cc

-} // namespace Nighthawk
+StatisticPtr CircllhistStatistic::combine(const Statistic& statistic) const {
+  auto combined = std::make_unique<CircllhistStatistic>();
+  const auto& b = dynamic_cast<const CircllhistStatistic&>(statistic);


Let's find a better (more expressive) variable name than just "b".

changed it to "stat".

mum4k · 2020-07-09T21:25:39Z

source/common/statistic_impl.cc

+StatisticPtr CircllhistStatistic::combine(const Statistic& statistic) const {
+  auto combined = std::make_unique<CircllhistStatistic>();
+  const auto& b = dynamic_cast<const CircllhistStatistic&>(statistic);
+  hist_accumulate(combined->histogram_, &this->histogram_, 1);


Without seeing the API of the histogram, it is hard to figure out what does the literal 1 stand for. Is there an useful name we can give it by either making it a constant or placing an inline comment?

// Either: const int good_name = 1; hist_accumulate(combined->histogram_, &this->histogram_, good_name); // Or: hist_accumulate(combined->histogram_, &this->histogram_, /* good_name = */ 1);

Thanks Jakub for the advice. Added an inline comment.

mum4k · 2020-07-09T21:26:25Z

source/common/statistic_impl.cc

+    return proto;
+  }
+
+  std::vector<double> quantiles{0,    0.1,   0.2,  0.3,   0.4,  0.5,   0.55,  0.6,


Can we add a comment indicating how did we select these quantile values?

mum4k · 2020-07-09T21:27:36Z

source/common/statistic_impl.cc

+    return proto;
+  }
+
+  std::vector<double> quantiles{0,    0.1,   0.2,  0.3,   0.4,  0.5,   0.55,  0.6,


(optional) If this and the next vector can be const, marking them so helps the reader understand that they aren't going to be modified further down.

Signed-off-by: Qin Qin <qqin@google.com>

mum4k

Thank you @qqustc, just last two nits.

mum4k · 2020-07-13T20:20:37Z

source/common/statistic_impl.h

+    return count() == 0 ? std::nan("") : hist_approx_stddev(histogram_);
+  }
+  StatisticPtr combine(const Statistic& statistic) const override;
+  uint64_t significantDigits() const override { return 1; }


This comment may have been missed. Even if we end up not moving this to the .cc file, we should add a comment explaining the choice of "1" as the precision.

mum4k · 2020-07-13T20:23:28Z

source/common/statistic_impl.h

+
+  const absl::optional<int> worker_id() { return worker_id_; }
+
+protected:


I apologize, I made a typo in the comment. I meant to say "rather than private". We should always prefer private methods and fields as they minimize the public API and allow us to execute faster (internal changes).

Can this be private instead?

oschaaf · 2020-07-13T21:55:23Z

source/common/statistic_impl.cc

+    return proto;
+  }
+
+  // List of quantiles is based on hdr_proto_json.gold which lists the quantiles provided by


Note: I think HdrHistogram will emit more percentiles as the number of (differently-valued) data points added grows. So it's not a fixed set with HdrHistogram.
I am not sure if there is an equivalent way to iterate the available buckets with libcircllhist to get more detail out, and I am also not sure this is really a problem, but I thought it would be good to call out the difference here.

(The only direct implication I figure it would have today is that the Fortio plots would be drawn with less data points)

Thanks Otto for the information! Updated the comment to just List of quantiles is based on hdr_proto_json.gold.
From the APIs in circllhist.h, there is no easy way to iterate the quantiles. If the listed data points turn out to be not enough in the future, we can always add more here.

Signed-off-by: Qin Qin <qqin@google.com>

qqustc · 2020-07-14T00:21:01Z

Thank you @qqustc, just last two nits.

Thank you @mum4k for the review.
Made scope_ private, exposed it through a public function method scope() for child class SinkableHdrStatistic to use.
Previously member scope_ is protected in SinkableStatistic, it is because scope_ is used in the child class SinkableHdrStatistic.

qqustc · 2020-07-14T01:56:55Z

/retest

repokitteh-read-only · 2020-07-14T01:56:58Z

🙀 failed invoking rebuild of ci/circleci: coverage: 403 Forbidden

🐱

Caused by: a #391 (comment) was created by @qqustc.

see: more, trace.

Signed-off-by: Qin Qin <qqin@google.com>

qqustc force-pushed the pr1 branch from 22a4ca5 to 2cae2d9 Compare July 1, 2020 02:41

qqustc requested a review from oschaaf July 1, 2020 02:52

qqustc added the waiting-for-review A PR waiting for a review. label Jul 1, 2020

oschaaf requested changes Jul 1, 2020

View reviewed changes

oschaaf added waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. and removed waiting-for-review A PR waiting for a review. labels Jul 1, 2020

qqustc and others added 8 commits July 6, 2020 18:43

Add new NH statistic class CircllhistStatistic and SinkableCircllhist…

af1ea54

…Statistic Signed-off-by: Qin Qin <qqin@google.com> Signed-off-by: qqustc <qqustc@gmail.com> Signed-off-by: Qin Qin <qqin@google.com>

Update statistic_impl.h

43d7ee2

Signed-off-by: Qin Qin <qqin@google.com>

Update statistic_impl.cc

77249b9

Signed-off-by: Qin Qin <qqin@google.com>

Update statistic_test.cc

bdecd69

Signed-off-by: Qin Qin <qqin@google.com>

Update circllhist_proto_json.gold

2c14561

Signed-off-by: Qin Qin <qqin@google.com>

Update statistic_impl.h

97ccc8a

Signed-off-by: Qin Qin <qqin@google.com>

Update statistic_test.cc

6876450

Signed-off-by: Qin Qin <qqin@google.com>

Refactor SinkableCircllhistStatistic

8fd0a05

Signed-off-by: qqustc <qqustc@gmail.com> Signed-off-by: Qin Qin <qqin@google.com>

qqustc force-pushed the pr1 branch from c43a746 to 8fd0a05 Compare July 6, 2020 22:44

fix clang_tidy

cd6b29e

Signed-off-by: Qin Qin <qqin@google.com>

qqustc added waiting-for-review A PR waiting for a review. and removed waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. labels Jul 7, 2020

Update statistic_test.cc

684fe0b

Signed-off-by: Qin Qin <qqin@google.com> Signed-off-by: qqustc <qqustc@gmail.com>

qqustc force-pushed the pr1 branch from 2a2f1c2 to 684fe0b Compare July 7, 2020 14:22

replace ASSERT with RELEASE_ASSERT

dc655a4

Signed-off-by: Qin Qin <qqin@google.com>

qqustc dismissed oschaaf’s stale review via a9d4d5d July 8, 2020 14:11

format

e4d6672

Signed-off-by: Qin Qin <qqin@google.com>

qqustc added waiting-for-review A PR waiting for a review. and removed waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. labels Jul 8, 2020

oschaaf previously approved these changes Jul 8, 2020

View reviewed changes

qqustc assigned mum4k Jul 8, 2020

mum4k requested changes Jul 9, 2020

View reviewed changes

mum4k added waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. and removed waiting-for-review A PR waiting for a review. labels Jul 9, 2020

update statistic_impl

db78892

Signed-off-by: Qin Qin <qqin@google.com>

qqustc dismissed oschaaf’s stale review via db78892 July 10, 2020 16:09

update statistic_impl.cc

8903b60

Signed-off-by: Qin Qin <qqin@google.com>

qqustc added waiting-for-review A PR waiting for a review. and removed waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. labels Jul 10, 2020

mum4k requested changes Jul 13, 2020

View reviewed changes

mum4k added waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. and removed waiting-for-review A PR waiting for a review. labels Jul 13, 2020

oschaaf reviewed Jul 13, 2020

View reviewed changes

update statistic_impl

2f0e580

Signed-off-by: Qin Qin <qqin@google.com>

qqustc added waiting-for-review A PR waiting for a review. and removed waiting-for-changes A PR waiting for comments to be resolved and changes to be applied. labels Jul 14, 2020

qqustc added 2 commits July 13, 2020 22:00

update comment

f01e18f

Signed-off-by: Qin Qin <qqin@google.com>

update statistic_impl

93bf7e8

Signed-off-by: Qin Qin <qqin@google.com>

mum4k approved these changes Jul 14, 2020

View reviewed changes

mum4k merged commit b4cebe4 into envoyproxy:master Jul 14, 2020

qqustc deleted the pr1 branch July 14, 2020 23:54

qqustc mentioned this pull request Jul 15, 2020

Add stats to record latency metrics for each response code in NH #381

Merged


		const absl::optional<int> worker_id() { return worker_id_; }

		protected:

Conversation

qqustc commented Jul 1, 2020

Uh oh!

qqustc commented Jul 1, 2020

Uh oh!

repokitteh-read-only bot commented Jul 1, 2020

Uh oh!

qqustc commented Jul 1, 2020

Uh oh!

oschaaf commented Jul 1, 2020

Uh oh!

repokitteh-read-only bot commented Jul 1, 2020

Uh oh!

oschaaf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqustc Jul 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oschaaf commented Jul 1, 2020

Uh oh!

oschaaf left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oschaaf commented Jul 2, 2020

Uh oh!

qqustc commented Jul 3, 2020 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqustc Jul 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

qqustc Jul 7, 2020 •

edited

Loading

oschaaf left a comment •

edited

Loading

qqustc Jul 10, 2020 •

edited

Loading

qqustc commented Jul 14, 2020 •

edited

Loading