Skip to content

ESQL: Add MV_PSERIES_WEIGHTED_SUM for score calculations used by security solution#109017

Merged
machadoum merged 28 commits intoelastic:mainfrom
machadoum:siem-ea-9521
Jul 31, 2024
Merged

ESQL: Add MV_PSERIES_WEIGHTED_SUM for score calculations used by security solution#109017
machadoum merged 28 commits intoelastic:mainfrom
machadoum:siem-ea-9521

Conversation

@machadoum
Copy link
Member

@machadoum machadoum commented May 24, 2024

Adds a MV_PSERIES_WEIGHTED_SUM which security solution will use.

This PR created MV_PSERIES_WEIGHTED_SUM function for converting sorted lists of numbers into a bounded score. It is a generalization of the riemann zeta function, but we don't name it because we don't support complex numbers and don't want to make folks think of mystical number theory things. This is just a weighted sum that is adjacent to magic.

A bit of math

Given the general form of p-series:
$$\sum_{n=1}^{\infty} \frac{1}{n^p} = \frac{1}{1^p} + \frac{1}{2^p} + \frac{1}{3^p} ...$$

We multiply every element on the input array by its corresponding term in the equation.

MV_PSERIES_WEIGHTED_SUM([70, 45, 21 , 21, 21], 3)
Where 3 is the p parameter and [70, 45, 21, 21, 21] is the input array

It does the following calculation:
$$(\frac{70}{1^3} + \frac{45}{2^3} + \frac{21}{3^3} + \frac{21}{4^3} + \frac{21}{5^3})$$

@elasticsearchmachine elasticsearchmachine added v8.15.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels May 24, 2024
@machadoum machadoum changed the title Siem ea 9521 Siem ea 9521 - RIEMANN_ZETA May 24, 2024
@machadoum machadoum changed the title Siem ea 9521 - RIEMANN_ZETA Siem ea 9521 - RIEMANN_ZETA POC May 29, 2024
@machadoum machadoum changed the title Siem ea 9521 - RIEMANN_ZETA POC Siem ea 9521 - PSeries Weighted Sum Jul 26, 2024
@nik9000 nik9000 changed the title Siem ea 9521 - PSeries Weighted Sum ESQL: Add MV_PSERIES_WEIGHTED_SUM for score calculations used by security Jul 26, 2024
@nik9000 nik9000 added >feature :Analytics/ES|QL AKA ESQL and removed external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 26, 2024
@machadoum machadoum changed the title ESQL: Add MV_PSERIES_WEIGHTED_SUM for score calculations used by security ESQL: Add MV_PSERIES_WEIGHTED_SUM for score calculations used by security solution Jul 26, 2024
@elasticsearchmachine
Copy link
Collaborator

Hi @machadoum, I've created a changelog YAML for you.

@machadoum machadoum self-assigned this Jul 26, 2024
@machadoum machadoum marked this pull request as ready for review July 26, 2024 13:32
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 26, 2024
@astefan astefan self-requested a review July 26, 2024 14:16
@nik9000
Copy link
Member

nik9000 commented Jul 26, 2024

I pushed a bit with @Fixed but there's a bug because the generated code uses p as a name and there's a clash.

@nik9000
Copy link
Member

nik9000 commented Jul 26, 2024

Fix for the problem with @Fixed: #111365 (review)

@nik9000
Copy link
Member

nik9000 commented Jul 26, 2024

Fix for the problem with @Fixed: #111365 (review)

Which I merged in.

Copy link
Contributor

@ivancea ivancea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@machadoum machadoum merged commit f79c621 into elastic:main Jul 31, 2024
@nik9000
Copy link
Member

nik9000 commented Jul 31, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >feature Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.16.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants