[APM]: Replace error occurrence watchers with Kibana Alerting by dgieselaar · Pull Request #46547 · elastic/kibana

dgieselaar · 2019-09-25T06:50:48Z

This is a proof of concept that replaces the APM error occurrence Watcher (an Elasticsearch feature) with the new Kibana alerting and actions plugin. The Slack action succesfully fires, but I haven't bothered with the Email action because the approach is pretty similar, and we decided to timebox this to two days.

Here's the gist:

Define an Alert Type for error occurrences. An Alert Type is essentially a function (called an executor) that, given a set of parameters, decides whether (and which) actions need to be triggered. Actions are triggered via an Alert Instance, which captures state of previous executions (for the purpose of this POC, I don't think we need that state).
When the user configures a new alert, we create an Alert object on the server. An Alert is a configuration object for an Alert Type, that tells it at what interval it and with what parameters it should execute.
An Alert also configures the action groups that can be triggered from the Alert Type executor. For this POC, we create Slack and Email actions based on the user's input when configuring an alert, and add them to the default group.
In the executor of the Alert Type, that now runs at the configured interval for the Alert that was created, we run a query for the number of error occurrences and determine whether the threshold was exceeded. If so, we fire the default actions for the Alert, which can be either the Slack or Email action, or both.
To make formatting easier, we pre-process some of the output from the query and pass this as context to the actions that are triggered.

To test this, make sure to explicitly enable both required plugins in your kibana config file:

xpack.actions.enabled: true
xpack.alerting.enabled: true

Notes:

Currently an Alert only supports interval as a scheduling option. That means that we cannot run the executor at a given time each day, which is something that our current implementation does support. I've been told expanding the scheduling options is on the roadmap.
To determine the timeframe for the query we run in the executor, we need the interval parameter. However, it doesn't seem like that's automatically available in the executor, so we pass it as a parameter instead. Maybe there's a nicer way to solve this.
We have to create actions in addition to each Alert that we create (AFAICT), because we allow our users to configure the Slack webhook URL and email addresses in the form. This could present some maintenance issues. It might make more sense to expect the user to create specific Slack/Email actions first, and then provide them with a UI to select those predefined actions. This would limit the amount of maintenance/cleanup of orphaned actions that would be needed in our current implementation.
The Alerting plugin still uses template rendering, but it's a lot easier now that we can pre-process the output from the query with JavaScript, and then pass it to a Mustache renderer. One thing that could me off guard was that Mustache escapes by default, which broke the formatting of the Slack message initially.

Some questions/suggestions for the Alerting/Actions team:

Some of the configuration options for the email and slack actions are now seemingly moved to the secrets param, but this is not reflected in the documentation (https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/actions/README.md)
As explained above, would be nice to have the interval available in the executor as well (if it's not already there and I missed it). Seems like it's a common use case.
I could not figure out what actionGroups is for in the Alert Type. It's seemingly not documented, but it is required.
The typings of registerType could be improved by having typed params (instead of Record<string, any>). Similar to what we did for APM in [APM] migrate to io-ts #42961 (happy to open a PR if welcome).
The kbn-action CLI tool has been super useful, thanks for taking the time to build that.

dgieselaar · 2019-09-25T06:51:30Z

Woops, this was supposed to be a draft PR. I'll update the description in a bit.

elasticmachine · 2019-09-25T07:59:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

dgieselaar · 2019-09-25T10:44:45Z

x-pack/legacy/plugins/apm/common/alerting/constants.ts

I don't think we need this right now. I was figuring out if we could create the email action we need when registering the alert type, but because I never got around to implementing email it's not being used.

dgieselaar · 2019-09-25T10:45:26Z

x-pack/legacy/plugins/apm/index.ts

I'm not sure if these should be required, or if they are optional. If it's the latter, not sure how we can get access to these plugins on startup.

Really good question for platform how to handle optional dependencies. I think it has come up before but can't remember the answer.

New platform allows for optional dependencies, but legacy platform does not. If you are depending on something in a legacy plugin that could be disabled, you need to make sure that you implement isEnabled and check that the dependency is enabled, otherwise Kibana will crash.

dgieselaar · 2019-09-25T10:46:07Z

...egacy/plugins/apm/public/components/app/ServiceDetails/ServiceIntegrations/WatcherFlyout.tsx

I've simplified this because we can just pass a csv to the action, because it uses nodemailer under the hood. Never tested it though.

dgieselaar · 2019-09-25T10:46:46Z

x-pack/legacy/plugins/apm/server/lib/alerting/error_occurrence/create_alert.ts

Not sure if we're allowed to do this or if we have to use the API.

This is ok to do, all the business logic currently lives within the client and you're getting the client from the request (which is good). You will just have to handle validation until we have some within the client (if we decided to do so).

dgieselaar · 2019-09-25T10:48:28Z

x-pack/legacy/plugins/apm/server/routes/error_occurrence_alert.ts

I've created a route for this, because we have to execute several concurrent and sequential requests, and the server is a more robust environment for that kind of dependencies.

mikecote · 2019-09-27T19:16:06Z

...acy/plugins/apm/server/lib/alerting/error_occurrence/register_error_occurrence_alert_type.ts

You can also use services.callCluster(...). It will be in the context of the user who created the alert (security wise). The approach you have is fine as well if you don't want that.

elasticmachine · 2019-10-02T13:24:58Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-11-06T13:05:33Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: c6b3361

ogupte · 2019-11-13T08:49:31Z

...egacy/plugins/apm/public/components/app/ServiceDetails/ServiceIntegrations/WatcherFlyout.tsx

-      .then((id: string) => {
+      .then(savedObject => {
        this.props.onClose();
+        const id = 'id' in savedObject ? savedObject.id : NOT_AVAILABLE_LABEL;


Maybe I'm outmoded, but isn't the in operator generally discouraged since it's at risk for prototype hijacking?

ogupte · 2019-11-13T08:53:22Z

x-pack/legacy/plugins/apm/server/lib/alerting/error_occurrence/create_alert.ts

+              '<br/>' +
+              '{errorLogMessage}<br/>' +
+              '{errorCulprit}<br/>' +
+              '{docCount} occurrences<br/>',


do want to preserve new lines \n characters in email template like in the slack template?

dgieselaar · 2020-03-19T07:49:47Z

Closing in favor of #59566.

dgieselaar requested a review from a team September 25, 2019 06:50

dgieselaar commented Sep 25, 2019

View reviewed changes

This was referenced Sep 26, 2019

[APM] Alerting POC #41032

Closed

[Observability] Alerting POC, metrics threshold alert #46511

Closed

mikecote reviewed Sep 27, 2019

View reviewed changes

dgieselaar force-pushed the alerting-poc branch from f01d96d to 087d6ec Compare October 2, 2019 12:46

Zacqary mentioned this pull request Oct 9, 2019

[Metrics] POC for Alerting on Metric Threshold #47165

Closed

smith added the WIP Work in progress label Oct 22, 2019

[APM]: Replace error occurrence watchers with Kibana Alerting

c6b3361

dgieselaar force-pushed the alerting-poc branch from 087d6ec to c6b3361 Compare November 6, 2019 12:22

ogupte reviewed Nov 13, 2019

View reviewed changes

andrewvc mentioned this pull request Nov 25, 2019

[Uptime] Alerting Implementation #51606

Closed

dgieselaar closed this Mar 19, 2020

dgieselaar deleted the alerting-poc branch March 19, 2020 07:50

Conversation

dgieselaar commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dgieselaar commented Sep 25, 2019

Uh oh!

elasticmachine commented Sep 25, 2019

💔 Build Failed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 2, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 6, 2019

💔 Build Failed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogupte Nov 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgieselaar commented Mar 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dgieselaar commented Sep 25, 2019 •

edited

Loading

ogupte Nov 13, 2019 •

edited

Loading