[APM] Add additional variables to the action variables available in APM threshold alert types

[formgeist]

Summary

Our current list of available variables that can be used to provide useful context to an action that will be sent/stored when an alert is triggered, could use some additional options.

Current list of APM specific variables for both transaction duration and error rate thresholds;

• {{context.serviceName}}
• {{context.transactionType}}

Solution

Extend the list of variables to include;

• {{context.value}}
• {{context.thresholdValue}}
• {{context.serviceEnvironment}}

Not sure if feasible at this time, but it would be great to be able to include a {{serviceLink}} that will allow the user to choose whether to go straight to the Service view or go to the Alert detail page in Alerting Mgmt.

cc @nehaduggal

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[DanRoscigno]

@formgeist , would you like me to poll the Elastic SREs and Infra folks for what they would put into alerts?

[formgeist]

@DanRoscigno That would be very useful, thank you!

[DanRoscigno]

Hi @formgeist , I received some feedback from @matschaffer :

Do we need a fixed list? I would hope that we're able to select any field from the returned payload.
We make fairly heavy use of ctx.payload on slack alerts today.

Would it be possible to access all fields? I understand that there is a tradeoff between flexibility and ease of use. Maybe a toggle to switch from a set list of fields to "everything" and a way to add a necessary field to the "set list".

[dgieselaar]

@DanRoscigno one of the issues here is that there's not a single document that triggers the alert, it's an aggregation on many documents that could all have different values. The watcher we currently have is per error group, so it makes sense to have a sample error there, but the new error rate alert is per service, so any error could have contributed to the threshold being exceeded.

[DanRoscigno]

I was not aware that the alerts for APM services would be based on aggregates of multiple documents. That certainly complicates things. Thanks for the background @dgieselaar

[matschaffer]

Thanks @DanRoscigno, @dgieselaar - as long as the limitation is specifically in the scope of APM-based alerts, I have no strong opinions. Guess we'll see how it shapes up :)

cc @tobio who has been doing more early work with kibana alerting and might have thoughts about how this applies to more generic alerting (e.g., on app logs or metricbeat time series)

[DuoLaAMengDianHuanXiang]

You can use watcher for email alert

[sorenlouv]

These will be available in 7.10.