Add the jaeger-mixin for monitoring

[gouthamve]

I'm submitting this after discussing with @jpkrohling. See other mixin usages here: https://github.com/etcd-io/etcd/tree/master/Documentation/etcd-mixin

This provides a customisable set of alerts and dashboards that can be used to monitor jaeger, in the mixin format that is popularized by the Prometheus community.

[codecov]

Codecov Report

Merging #1668 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1668   +/-   ##
=======================================
  Coverage   98.51%   98.51%           
=======================================
  Files         193      193           
  Lines        9314     9314           
=======================================
  Hits         9176     9176           
  Misses        110      110           
  Partials       28       28

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dcaea9d...a30b0a0. Read the comment docs.

[jpkrohling]

Another question: should we provide a pre-built version of the alerts and dashboards, for users who don't want/can't install jsonnet and/or jb?

[yurishkuro]

Not at all to diminish the value of this PR, but Is there a UI that can edit the dashboard? I can tolerate editing alert definitions manually (even then - bleh), but editing dashboard files is beyond poor user experience. At Uber we take the approach that you have a UI for editing these complex configs, but the UI generates a code diff in the end, so that we can treat configuration as code, go through code review, etc.

About the alert file, does the format allow including some descriptions or links to run books? I find it very bad practice to have alerts that only include some cryptic “name”, esp. if someone gets woken up by them in the middle of the night. If we know enough about a condition to decide that it warrants an alert, then alert should include that explanation, and, preferably, some pointers to remediation steps.

[jpkrohling]

Is there a UI that can edit the dashboard?

The dashboards can indeed be changed directly in Grafana, once they are loaded there. Note however that we'd either need to provide the dashboard JSON files generated from the mixins, or have users generate the JSON by themselves.

About the alert file, does the format allow including some descriptions or links to run books?

Good point -- I don't know the answer myself, but I think we could extend our documentation in the future to include a description about those alerts.

[gouthamve]

How to mix runbooks for mixins is an ongoing discussion in the mixin community and there will soon be a way to distribute the runbooks too :)

…

On Thu, Jul 18, 2019, 8:17 PM Juraci Paixão Kröhling < ***@***.***> wrote: Is there a UI that can edit the dashboard? The dashboards can indeed be changed directly in Grafana, once they are loaded there. Note however that we'd either need to provide the dashboard JSON files generated from the mixins, or have users generate the JSON by themselves. About the alert file, does the format allow including some descriptions or links to run books? Good point -- I don't know the answer myself, but I think we could extend our documentation in the future to include a description about those alerts. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1668?email_source=notifications&email_token=ABYDOH66DDXGFI2WXSLLSP3QAB7BPA5CNFSM4IEZLXE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2IW7OY#issuecomment-512847803>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYDOH6LYE3VVJCKAP2YMVDQAB7BPANCNFSM4IEZLXEQ> .

[yurishkuro]

How to mix runbooks for mixins is an ongoing discussion in the mixin community and there will soon be a way to distribute the runbooks too

I was thinking of something simpler, a description field that can contain the explanation of the meaning of the alert and suggestions for remediation, or just a link to a doc (the same explanations would make sense to exist in the documentation).

[jpkrohling]

@gouthamve do you have any news on this one?

[jpkrohling]

@gouthamve I just opened #1700 with the changes discussed here. If you had them implemented already, let me know and I'll close my PR in favor of yours. Otherwise, feel free to close this one and comment on that one. I tried to keep you as the author of the original commit.

[jpkrohling]

@objectiser, @yurishkuro, could you please review again? Once this is merged, I'll try to get feedback from people already using Prometheus to monitor Jaeger to see if they can help us improve the mixins.

[jpkrohling]

@yurishkuro, @objectiser is there anything else pending, or can this one be merged?

[objectiser]

Would be good to get an answer about the annotations - but that isn't necessarily a blocker. We could evolve this area with feedback from the community.

[yurishkuro]

I don't have objections to merging. This seems highly experimental, so best to get feedback. One general comment is I think the files could use some comments, e.g. when I think back to sample config files for Cassandra or httpd, every property there is documented.

[jpkrohling]

Rebased, to see if we can get past of the build failures (master now seems to have travis_retry in key steps)