[RFC] OpenSearch Events Correlation Engine

[sbcd90]

Problem Statement

OpenSearch is a scalable, flexible, and extensible open-source software suite for search, analytics, and observability applications licensed under Apache 2.0.
OpenSearch includes a data store and search engine where customers can store their business, operational, and security data from a variety of sources & run search queries on them.

Since the various customer infrastructure events, such as security events, observability events etc, spans across multiple indices & data streams, a strong correlation across these indices (or data streams) helps customers to identify patterns and dive into the relationship of events occurring across different systems in their infrastructure.

Definitions

Events Correlation Engine

Correlation Engine is an Events Knowledge Graph which can be used to identify and store connected events data spanning across multiple indices or data streams. Also, it helps generate insights by correlating the recent/historical data based on time windows provided by the client .

The Events Correlation Engine provides an approach to help customers correlate events across log sources by allowing customers to define their own Correlation Rules exactly once, while then generating correlations between events from different log sources automatically.

Dimensions of Correlation

Time Window

Time Window is the most basic Dimension of Correlation that can be defined by the user. Correlation Engine would show all possible correlations across all indices within the specified time window if no other dimension is provided.

Source Events Indices/DataStreams

While Time Window is an important dimension of correlation, users also need to provide source events indices(or datastreams) on which Correlation rules can be defined which acts as an additional dimension of correlation.

Query Language for Correlation Rules

The most granular level of correlation supported by the Correlation Engine is using correlation rules or queries over the source events indices or datastreams. These rules allow the Correlation Engine to eliminate false positives & present to the user a list of highly accurate correlated search results.

One of the popular choices for defining Correlation Rules is Event Query Language(EQL) from Elasticsearch. EQL supports ECS today.

Here is a sample EQL based Correlation Query.

{
  "query": """
      [network where src_addr == "4.5.6.7" and severity_id = -1]
      [ad_ldap where ResultType == 50126]
      [windows where host.hostname == "EC2AMAZ*"]
      [others_application where StatusCode == 403]
      [s3 where aws.cloudtrail.eventName="ReplicateObject"]
  """
}

High-Level Design

There are 2 high level components in the design of Events Correlation Engine .

Correlation Query Service

This sub-system manages the lifecycle of the Correlation Rules created by the users. Users can create, update, read or delete rules using the REST apis provided by this layer.

The language for defining Correlation Rules is still not finalized. EQL is one of the examples for defining Correlation Rules.

Correlation Service

The internals of the Correlation Engine is composed of 4 major components.

• HNSW Graph based vector storage - this is HNSW Graph based storage used to store all event vectors & query them at the vector level.
• Insertion Handler - the most important piece of the Correlation Engine is its insertion algorithm. In this layer, events are converted to k-dimensional vectors & are stored in the vector storage layer mentioned above along with their correlations.
• Search Handler - the second most important piece of the Correlation Engine allows user to specify a particular event, & then converts it to a k-dimensional vector & then uses it to query its neighboring eventswhich are actually its correlated events within a time window.
• Join Handler - the Join task determines immediate neighbors of a particular event, given the correlation rules defined by the user for the log indices(or datastreams) they wish to correlate .

Use Cases

Security Analytics Correlation Engine for correlating security events

Security Analytics is an open-source solution for security operations in OpenSearch. Security Analytics’ threat detection engine converts the detection rules into executable OpenSearch queries which are then matched against the logs or events ingested by the user to generate findings. The trigger condition filters are further applied on the findings to generate alerts.

Today in Security Analytics, the generated findings belong to individual log types & there is no way to automatically correlate between them. Users would manually need to browse through the findings generated for individual log categories & then need to identify patterns manually.

The Security Analytics Correlation Engine provides an approach to solve this issue by allowing the customers to define the correlation metadata across log categories exactly once & then generating correlations between findings from different log categories automatically.

Here is link to RFC

[praveensameneni]

Thank you for the proposal @sbcd90 . I think correlation is a great primitive that can be leveraged across different use cases of Security Threat detection, Observability and generic log analytics correlation. Would love to get feedback from the community.

cc: @nknize , @dblock , @CEHENKLE

[getsaurabh02]

@sbcd90 Thanks for creating the RFC. This is a great start.

Adding some more thoughts to the What, How and Why part of a generic Correlation Engine Framework that we are thinking here:

What is Event Correlation
Event correlation automates the process of analyzing the findings (patterns) from the documents which came from the various log sources, to detect incidents and problems with deeper insights and patterns. Using an event correlation helps identify relationships between the events that has occurred now and its previous instances. It also helps to identify relationship from the other log sources based on the common field vectors (such as IP) which otherwise would have gone unnoticed.

How does an Event Correlation work:
Writing correlation rules/queries repeatedly is time consuming process for operators and requires a deep understanding of how logs are related and structured. The solution should provide ability to automate some of these steps with a background job, so that relationsal insights can be generated and persisted once the data is available, using the pre-authored rules or criteria:

• Event Aggregation: This process encompasses gathering the related findings data from different log sources, based on time window of occurrences, into a single location, if necessary.
• Event Filtering: Finding events then needs to filtered by pre-packaged or user-defined criteria such as source, timeframe or event level. This step may alternately be performed before aggregation, but is generally more accepted within the post aggregation step.
• Event Deduplication: This step identifies and dedupe events that are occurrences of the same issue. Say 1,000 failed login attempts were reported over two hours. Rule matching would generate 1,000 alerts, but there aren’t 1,000 problems. In reality, there were 1,000 instances of one issue, where all these attempts were from a single IP within time window which of interest.
• Event Normalization: The normalization step ensures findings data collected from many different sources is put into a single consistent format that then allows the solution to correlate this data. For example, one findings data calls something as a “host” and another one calls them “instances” based on the log sources Rule Schema. Normalization may use “Affected Device” to refer to the contents of the fields so that, the solution interprets it the same way.
• Scoring: The most complex step of the process, event interdependencies are finally analyzed to determine the relevance and corresponding scoring. This also helps identify he depth and width of the related threat (neighbors). Such as events on one device are examined to determine its impact on every device in the network.

Why Event Correlation - Use-Cases and Examples

• User Behavior Detection A graph based data model can be used to uncover the behavioral patterns in real time and prevent confidential information being stolen. Such as tracing an error / alert / problem back to its source where individual events otherwise could go unnoticed.
  • A file could be corrupted while someone was attempting to write to it and an alert was generated.
  • A high CPU usage alert that was received when a user was connecting to it.
  • An IP identified under multiple failed login attempts from the kernel logs of host, can be mapped against the port scan events on the same or connected host. The same pattern can also be used for comparing other port scan events from the past.
  • A user plugging in a new mobile disk to the host, copying a file and then removing the mobile disk.
• Threat Detections - Correlations can be used to trace these alerts back to a user and even to a specific IP addresses. Doing so successfully requires traversing multiple hops, something that can be efficiently done with a connected data models.
  • An IP identified under network DoS attack due to high request rate, can be run against the application logs to identify the kind of object accessed (such as high_null_records_requests), which in isolation would have gone unnoticed.
  • A user with less privileged role bypassing a firewall check and then attempting to read from a restricted file.
• Anomaly Detection – The correlated events which models normal patterns of behavior, can also detect anomalous events in real time by identifying relationships and connections.
  • Flooding event detection when a service receives many more requests than usual – or a foot printing detection event when a service receives a large amount of requests from a single user, who may be probing for weaknesses in the security measures of that service.
  • Extracting feature sets can be used for machine learning models, such as finding IP addresses which are connected to blacklisted IP, already identified as a threat earlier. These connections could be measured in form of distances in the connected data models, such as one hop, two hop etc. based on the common linkages they share between them.

cc: @nknize @dblock

[nknize]

This is great! 💯 agree this should be a core feature. Lets progress not perfection this. A few initial questions / clarifications:

1. Any opposition to starting as a plugin? We can always decide to move it as a module later if that's the direction the community wants to take it.
2. What's the initial user facing API? The description above mentions EQL which is for threat modeling. Since this engine will support more use cases can we start w/ a simple DSL API and follow on w/ other languages on a per use case basis? What's that DSL API look like?
3. What's the surface area change to :server or any other modules or libraries? Depending on this we can decide to forgo feature flagging and just use @opensearch.experimental tags since plugins are optional and not installed by default. No need to shove behind another gate.

[sbcd90]

hi @nknize , Thanks a lot for reviewing the RFC. Here are my answers.

1. I agree with the proposal of starting it as a plugin & then moving it as a module later. @getsaurabh02 any thoughts on this?

2. I have added the api doc with this response below. The api doc has 2 sections. the first section is for the core plugin & the second section is specific to security-analytics UI mockups. We intend to extend the Correlation Engine plugin in core to security-analytics & add security-analytics specific apis in it.

3. There are no changes required in the :server or any other modules or libraries. We can then just use @opensearch.experimental tags.

Here is the api doc.

Core Plugin apis

Create Correlation Rule for an index/data stream

POST /_plugins/_correlation/rules

Request Body: 
{
  "correlate": [
    {
      "index": "vpc_flow",
      "query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6"
    },
    {
      "index": "windows",
      "query": "winlog.event_data.SubjectDomainName:NTAUTHORI*"
    },
    {
      "index": "ad_logs",
      "query": "ResultType:50126"
    },
    {
      "index": "app_logs",
      "query": "endpoint:/customer_records.txt"
    }
  ]
}

List Correlations for an event stored in an index/data stream

GET /_plugins/_correlation/events?event_id=425dce0b-f5ee-4889-b0c0-7d15669f0871
&index=ad_logs&nearby_events=20&time_window=10m

Response:

{
  "events": [
    {
      "event": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
      "index": "app_logs",
      "score": 0.000015182109564193524
    },
    {
      "event": "2485b623-6573-42f4-a055-9b927e38a65f",
      "index": "ad_logs",
      "score": 0.000001615897872397909
    },
    {
      "event": "051e00ad-5996-4c41-be20-f992451d1331",
      "index": "windows",
      "score": 0.000016230604160227813
    },
    {
      "event": "f11ca8a3-50d7-4074-a951-51439aa9e67b",
      "index": "s3",
      "score": 0.000001759401811796124
    },
    {
      "event": "9b86980e-5fb7-4c5a-bd1b-879a1e3baf12",
      "index": "vpc_flow",
      "score": 0.0000016306962606904563
    },
    {
      "event": "e7dea5a1-164f-48f9-880e-4ba33e508713",
      "index": "vpc_flow",
      "score": 0.00001632626481296029
    }
  ]
}

Security Analytics Plugin Apis

Create Correlation Rules between Log Types

POST /_plugins/_security_analytics/correlation/rules

Request Body:
{
  "correlate": [
    {
      "index": "vpc_flow",
      "query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6",
      "category": "network"
    },
    {
      "index": "windows",
      "query": "winlog.event_data.SubjectDomainName:NTAUTHORI*",
      "category": "windows"
    },
    {
      "index": "ad_logs",
      "query": "ResultType:50126",
      "category": "ad_ldap"
    },
    {
      "index": "app_logs",
      "query": "endpoint:/customer_records.txt",
      "category": "others_application"
    }
  ]
}

List all findings & their correlations within a time window

POST /_plugins/_security_analytics/correlate/findings/_search?
time_window_start=2023-03-24T00:00:00Z&time_window_end=2023-03-26T00:00:00Z

Request Body:
{
    "from" : 20,
    "size": 10,
    "query": {
        "match": {
            "logType": "windows"
        }
    }
}

Response:
{
  "findings": [
    {
      "finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
      "detector_type": "others_application",
      "correlated_findings": [{
        "finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
        "detector_type": "ad_ldap"
      }, {
        "finding": "f11ca8a3-50d7-4074-a951-51439aa9e67b",
        "detector_type": "s3"
      }]
    },
    {
      "finding": "f11ca8a3-50d7-4074-a951-51439aa9e67b",
      "detector_type": "s3",
      "correlated_findings": [{
        "finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
        "detector_type": "others_application"
      }]
    }
  ]
}

List correlations for a finding belonging to a log type

GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871
&detector_type=ad_ldap&nearby_findings=20&time_window=10m

Response:
{
  "findings": [
    {
      "finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
      "detector_type": "others_application",
      "score": 0.000015182109564193524
    },
    {
      "finding": "2485b623-6573-42f4-a055-9b927e38a65f",
      "detector_type": "ad_ldap",
      "score": 0.000001615897872397909
    },
    {
      "finding": "051e00ad-5996-4c41-be20-f992451d1331",
      "detector_type": "windows",
      "score": 0.000016230604160227813
    },
    {
      "finding": "f11ca8a3-50d7-4074-a951-51439aa9e67b",
      "detector_type": "s3",
      "score": 0.000001759401811796124
    },
    {
      "finding": "9b86980e-5fb7-4c5a-bd1b-879a1e3baf12",
      "detector_type": "network",
      "score": 0.0000016306962606904563
    },
    {
      "finding": "e7dea5a1-164f-48f9-880e-4ba33e508713",
      "detector_type": "network",
      "score": 0.00001632626481296029
    }
  ]
}

[getsaurabh02]

Agree with @sbcd90 on moving ahead with the suggested plan here and starting as a core plugin. It is also the quickest path forward, without requiring any changes in the server upfront. Also, it allows us to keep the changes isolated and sand boxed from performance side of view.

[dblock]

What's the tl;dr of why this feature needs to be in core, especially that it's going to start as _plugins/_correlation? Can it begin as an external plugin?

[getsaurabh02]

Thanks @dblock for the review. The Correlation Engine we are proposing here aims to provide the capability to build Events Knowledge Graph within the OpenSearch data set, which can be used to identify and store connected data events, possibly spanning across multiple indices or data streams. These knowledge graphs can further help generate insights by correlating the recent or historical data across custom time windows which users can provide.

Since it provides an approach to help users correlate events across log sources, while allowing them to define their own correlation Rules, the framework itself can be leveraged by different end user plugins to solve different end use-cases such as those related to Security Analytics, Observability, geospatial or trace analytics.

Going forward, once we have had baked the feature well as the core plugin, we will have further aim to provide the capability as the core module itself.

[YANG-DB]

This looks very interesting and has great potential , few point to continue the discussion

• why not use PPL as the correlation language ?

• as part of simple schema for Observability we are also introducing the correlation concepts as an ability to create relationships between different entities. This metadata schema information can be used to drive additional entity based correlations

• Observability signals correlation:
  

I'll be happy to discuss more on this Knowledge -Graph !!

Adding the correlation metadata Knowledge into the field mapping API

• [Feature request] Add field mapping correlation type metadata concept  #7082