[Security Solution][Entity Analytics][Risk Scoring] Add Risk Scoring Alert Filtering - Backend Implementation

[abhishekbhatia1710]

🎯 Summary

This PR implements the backend infrastructure for Risk Scoring Alert Filtering, enabling users to apply entity-specific KQL filters during risk score calculations. This enhancement allows for more targeted risk scoring by filtering alerts based on custom criteria.

🚀 What's New

Core Features

• Entity-Specific Filtering: Apply KQL filters to specific entity types (host, user, service)
• Backward Compatibility: Existing configurations continue to work without changes
• Graceful Error Handling: Invalid KQL filters are silently ignored to prevent query failures
• Migration Support: Automatic migration of existing saved objects to include new filters field

Technical Implementation

1. Saved Object Schema Enhancement

• Added filters field to risk-engine-configuration saved object
• Implemented migration logic (version 3) for existing configurations
• Updated mappings version to 5

2. Enhanced Risk Scoring Logic

• Created buildFiltersForEntityType helper function for entity-specific filter construction
• Integrated KQL parsing using @kbn/es-query utilities
• Applied filters at the aggregation level for optimal performance

3. API Endpoint Updates

• Enhanced configuration endpoint to accept filters parameter
• Updated preview endpoint to support filter testing
• Maintained backward compatibility with existing API contracts

🏗️ Architecture

graph TD
    A[User Configuration] --> B[Risk Engine Saved Object]
    B --> C[buildFiltersForEntityType]
    C --> D[KQL Parser]
    D --> E[Elasticsearch Query]
    E --> F[Risk Score Aggregation]
    F --> G[Filtered Results]
    
    H[API Request] --> I[Route Handler]
    I --> J[Risk Score Service]
    J --> K[calculateRiskScores]
    K --> C

Loading

🔧 Filter Processing Flow

sequenceDiagram
    participant U as User
    participant API as API Endpoint
    participant S as Risk Score Service
    participant F as Filter Builder
    participant ES as Elasticsearch
    
    U->>API: POST /api/risk_scores/preview
    API->>S: calculateScores(filters)
    S->>F: buildFiltersForEntityType()
    F->>F: Parse KQL filters
    F->>F: Build ES queries
    F-->>S: Return filters array
    S->>ES: Execute aggregation with filters
    ES-->>S: Return filtered results
    S-->>API: Return risk scores
    API-->>U: Return response

Loading

🧪 Testing

Unit Tests Added

• buildFiltersForEntityType function tests
• Saved object migration tests
• API endpoint filter parameter tests
• Error handling for invalid KQL filters

Test Coverage

• ✅ Entity-specific filter application
• ✅ Multiple filters for same entity type
• ✅ Empty filter arrays handling
• ✅ Invalid KQL filter graceful handling
• ✅ Backward compatibility verification

📋 API Testing

1. Configure Risk Engine with Filters

curl -X PUT "http://localhost:5601/api/risk_score/engine/saved_object/configure" \
  -H "Content-Type: application/json" \
  -H "kbn-xsrf: true" \
  -H "Authorization : Basic ***" \
  -d '{
    "filters": [
      {
        "entity_types": ["host"],
        "filter": "agent.type: filebeat"
      },
      {
        "entity_types": ["user"],
        "filter": "user.name: ubuntu"
      }
    ]
  }'

Expected Response:

{
  "risk_engine_saved_object_configured": true
}

Get risk engine saved object configuration:

curl --location --request GET 'http://localhost:5601/api/saved_objects/_find?type=risk-engine-configuration' \
--header 'elastic-api-version: 1' \
--header 'kbn-xsrf: true' \
--header 'x-elastic-internal-origin: true' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ***' \
--data '{
  "range": {
    "start": "now-30d",
    "end": "now"
  },
  "includeClosedAlerts": false
}'

Expected Response:

{
    "page": 1,
    "per_page": 20,
    "total": 1,
    "saved_objects": [
        {
            "type": "risk-engine-configuration",
            "id": "36e6f63d-e5d0-4919-8d41-4988dd708754",
            "namespaces": [
                "default"
            ],
            "attributes": {
                "dataViewId": ".alerts-security.alerts-default",
                "enabled": false,
                "filter": {},
                "interval": "1h",
                "pageSize": 3500,
                "range": {
                    "start": "now-30d",
                    "end": "now"
                },
                "excludeAlertStatuses": [
                    "closed"
                ],
                "_meta": {
                    "mappingsVersion": 5
                },
                "filters": [
                    {
                        "entity_types": [
                            "host"
                        ],
                        "filter": "agent.type: filebeat"
                    },
                    {
                        "entity_types": [
                            "user"
                        ],
                        "filter": "user.name: ubuntu"
                    }
                ]
            },
            "references": [],
            "managed": false,
            "migrationVersion": {
                "risk-engine-configuration": "10.3.0"
            },
            "updated_at": "2025-09-29T06:52:46.483Z",
            "created_at": "2025-09-26T08:42:58.037Z",
            "version": "WzExLDFd",
            "coreMigrationVersion": "8.8.0",
            "typeMigrationVersion": "10.3.0",
            "score": 0
        }
    ]
}

2. Preview Risk Scores with and without Filters

I added 5 alerts for this test
host.name : "pessimistic-permafrost.name"
host.name : "yellowish-minority.info"
user.name : "Roscoe_Stehr-Murazik"
user.name : "Sheridan_MacGyver55"
user.name : "ubuntu"

Without filters

curl --location 'http://localhost:5601/internal/risk_score/preview' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: true' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ****' \
--data '{
"data_view_id": ".alerts-security.alerts-default",
  "range": {
    "start": "now-30d",
    "end": "now"
  }}' | jq '{
  hosts: [.scores.host[] | {id_field, id_value}],
  users: [.scores.user[] | {id_field, id_value}]
}'

{
  "hosts": [
    {
      "id_field": "host.name",
      "id_value": "pessimistic-permafrost.name"
    },
    {
      "id_field": "host.name",
      "id_value": "yellowish-minority.info"
    }
  ],
  "users": [
    {
      "id_field": "user.name",
      "id_value": "Roscoe_Stehr-Murazik"
    },
    {
      "id_field": "user.name",
      "id_value": "Sheridan_MacGyver55"
    },
    {
      "id_field": "user.name",
      "id_value": "ubuntu"
    }
  ]
}

With user filter

curl --location 'http://localhost:5601/internal/risk_score/preview' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: true' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ****' \
--data '{
"data_view_id": ".alerts-security.alerts-default",
  "range": {
    "start": "now-30d",
    "end": "now"
  },
  "filters": [
      {
        "entity_types": ["user"],
        "filter": "user.name: ubuntu"
      }
    ]
}' | jq '{
  hosts: [.scores.host[] | {id_field, id_value}],
  users: [.scores.user[] | {id_field, id_value}]
}'

{
  "hosts": [
    {
      "id_field": "host.name",
      "id_value": "pessimistic-permafrost.name"
    },
    {
      "id_field": "host.name",
      "id_value": "yellowish-minority.info"
    }
  ],
  "users": [
    {
      "id_field": "user.name",
      "id_value": "Roscoe_Stehr-Murazik"
    },
    {
      "id_field": "user.name",
      "id_value": "Sheridan_MacGyver55"
    }
  ]
}

With user and host filter

curl --location 'http://localhost:5601/internal/risk_score/preview' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: true' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ****' \
--data '{
"data_view_id": ".alerts-security.alerts-default",
  "range": {
    "start": "now-30d",
    "end": "now"
  },
  "filters": [
      {
        "entity_types": ["user"],
        "filter": "user.name: ubuntu"
      },
      {
        "entity_types" : ["host"],
        "filter": "agent.type: filebeat"
      }
    ]
}' | jq '{
  hosts: [.scores.host[] | {id_field, id_value}],
  users: [.scores.user[] | {id_field, id_value}]
}'

{
  "hosts": [
    {
      "id_field": "host.name",
      "id_value": "pessimistic-permafrost.name"
    }
  ],
  "users": [
    {
      "id_field": "user.name",
      "id_value": "Roscoe_Stehr-Murazik"
    },
    {
      "id_field": "user.name",
      "id_value": "Sheridan_MacGyver55"
    }
  ]
}

3. Test Invalid KQL Filter Handling

curl -X POST "http://localhost:5601/api/risk_scores/preview" \
  -H "Content-Type: application/json" \
  -H "kbn-xsrf: true" \
  -d '{
    "data_view_id": "security-solution-default",
    "range": {
      "start": "now-30d",
      "end": "now"
    },
    "filters": [
      {
        "entity_types": ["host"],
        "filter": "invalid kql syntax {"
      }
    ]
  }'

Expected Response: Should return results without the invalid filter applied (graceful degradation)

🔄 Migration Strategy

• Automatic Migration: Existing saved objects are automatically migrated to include empty filters array

🎯 User Experience Impact

Before

• Risk scoring applied to all alerts without filtering
• Limited control over which alerts contribute to risk scores
• No way to focus on specific environments or conditions

After

• Targeted Risk Scoring: Filter alerts by environment, agent type, user groups, etc.
• Flexible Configuration: Apply different filters to different entity types

🔍 Key Technical Decisions

1. Entity-Specific Filtering: Filters are applied per entity type, allowing granular control
2. KQL Integration: Leverages existing Kibana Query Language for consistency
3. Aggregation-Level Filtering: Filters applied at ES aggregation level for performance

📝 Related Issues

• Closes: elastic/security-team#13606
• Related: elastic/kibana#224229

🚧 Next Steps

This PR implements the backend infrastructure. The frontend UI implementation will follow in a separate PR to:

• Add filter configuration UI components
• Integrate with existing risk engine configuration page
• Provide filter testing and validation features

---

Note: This is a backend-only implementation. Frontend changes will be delivered in a subsequent PR.

[elasticmachine]

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

[jloleysens]

Addition of optional filters field + mappings LGTM

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[CAWilson94]

Code Review only: LGTM

[tiansivive]

@elasticmachine merge upstream

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: eab481d
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-235770-eab481d570d9

Failed CI Steps

• FTR Configs #30

Test Failures

• [job] [logs] FTR Configs #30 / Core Analysis - Entity Store @ess @skipInServerlessMKI Entity Store APIs get and list "before all" hook in "get and list"

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.0MB	11.0MB	+309.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	695	694	-1

Total ESLint disabled count

id	before	after	diff
securitySolution	798	797	-1

History

• 💔 Build #356035 failed 68f14ee
• 💔 Build #355778 failed e2ac559
• 💔 Build #355680 failed e1d1610
• 💔 Build #355661 failed 70d2b3f
• 💔 Build #355587 failed 5e77a2d

cc @tiansivive @abhishekbhatia1710