poc(related) alerts custom score query

[kdelemme]

This PoC reuses the alert query hook and a custom table to display the results.
The query is currently not configurable by the user, but ideally we would offer some weight configuration on the different heuristics we use, as well as some more "on/off" filter, like on the alert status, alert start date, etc...

Ideally we could add to existing/new cases, and go to the alert details (or open in flyout)

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!

[kdelemme]

💬 Add the user's specified weights & filters in the mix

[tveasey]

Overall I like the approach (and it is really crafty to do it with custom scoring function 👍. I proposed some potential small tweaks. Happy to discuss these offline.

[tveasey]

If either set is empty or the intersect is empty we'll never get a match (the score is forced to zero). I'm not sure this is desirable.

For example, if $A$ is empty I would argue the instance on the candidate alert carries no additional information, i.e. the multiplier should be 1. Although I'm not sure under what circumstances this happens. Then if there is a match there is a positive boost, a boost of $1 + w \frac{A \cap B}{A \cup B}$ could be an option.

If both alerts have instances and there is no match I'm not sure, but I would we be inclined to think it should still be possible to match on other attributes. Perhaps though this should actively count against matching, i.e. multiplicative boost should be < 1.

One option which achieves all this is the following function:

$$1 + w \frac{A \cup B}{A \cap B} - 0.5 \times 1[A\neq \emptyset \text{ AND } B\neq \emptyset]$$

Here, $1[\cdot]$ denotes the indicator function. So the possible scores would be $0.5$, $1$, $1 + w \frac{m}{n}$ for $m \leq n$. Here, $w$ can be adjusted based on how much weight you want to assign uniquely matching the instance.

[tveasey]

A similar comment applies regarding no matching tags to no matching instances.

[tveasey]

I feel like exponential decay is reasonable. I think this should be based on the interval distance though (so might need script score). Specifically, if alerts have intervals $[a_1, b_1]$ and $[a_2, b_2]$ I would use something like

$$score = \exp(-(\max(a_1 - b_2, 0) - \max(a_2 - b_1, 0)) / T)$$

This calculates the distance between the intervals if they don't overlap and is otherwise 0.

On efficiency grounds one should probably add some filter conditions as well, i.e. must not "end time of one alert" << "start time of other". This will allow Lucene to skip many candidate matches and only run the script scoring on the remainder.