Entity Analytics 9.4 updates for UEBA integrations

packages/ded/changelog.yml

@@ -1,3 +1,8 @@
+- version: "3.0.0"
+  changes:
+    - description: Adds new fields for entity resolution with Entity Analytics in Elastic Stack 9.4. Includes new ML jobs, transforms, and dashboards.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/17626
 - version: "2.4.1"
   changes:
     - description: Update package docs with customization steps for ML jobs and transforms

packages/ded/docs/README.md

@@ -10,22 +10,22 @@
 - [Detect data exfiltration activity with Kibana’s new integration](https://www.elastic.co/blog/detect-data-exfiltration-activity-with-kibanas-new-integration)
 
 ## Installation
-1. **Upgrading**: If upgrading from a version below v2.0.0, see the section v2.0.0 and beyond.
+1. **Upgrading**: If upgrading from a version below v3.0.0, see the section v3.0.0 and beyond.
 1. **Add the Integration Package**: Install the package via **Management > Integrations > Add Data Exfiltration Detection**. Configure the integration name and agent policy. Click Save and Continue. (Note that this integration does not rely on an agent, and can be assigned to a policy without an agent.)
 1. **Install assets**: Install the assets by clicking **Settings > Install Data Exfiltration Detection assets**.
-1. **Check the health of the transform**: The transform is scheduled to run every 30 minutes. This transform creates the index `ml_network_ded-<VERSION>`. To check the health of the transform go to **Management > Stack Management > Data > Transforms** under `logs-ded.pivot_transform-default-<FLEET-TRANSFORM-VERSION>`. Follow the instructions under the header `Customize Data Exfiltration Detection Transform` below to adjust filters based on your environment's needs.
-1. **Create data views for anomaly detection jobs**: This package contains anomaly detection jobs that work on network events (e.g. `logs-endpoint.events.network-*`) and file events (`logs-endpoint.events.file-*`) respectively. See the _Anomaly Detection Jobs_ section below for more details. _Tip: If you only have one of the above data sources (network or file), you can only follow the steps pertaining to that index._ A separate designated index (`ml_network_ded.all`) collects network logs from a transform. Before enabling the anomaly detection jobs, create a data view with both index patterns.
+1. **Check the health of the transform**: The transform is scheduled to run every 30 minutes. This transform creates the index `ml_network_ded_ea-<VERSION>`. To check the health of the transform go to **Management > Stack Management > Data > Transforms** under `logs-ded.pivot_transform_ea-default-<FLEET-TRANSFORM-VERSION>`. Follow the instructions under the header `Customize Data Exfiltration Detection Transform` below to adjust filters based on your environment's needs.
+1. **Create data views for anomaly detection jobs**: This package contains anomaly detection jobs that work on network events (e.g. `logs-endpoint.events.network-*`) and file events (`logs-endpoint.events.file-*`) respectively. See the _Anomaly Detection Jobs_ section below for more details. _Tip: If you only have one of the above data sources (network or file), you can only follow the steps pertaining to that index._ A separate designated index (`ml_network_ded_ea.all`) collects network logs from a transform. Before enabling the anomaly detection jobs, create a data view with both index patterns.
     1. Go to **Stack Management > Kibana > Data Views** and click **Create data view**.
-    1. Enter the name of your respective index patterns in the **Index pattern** box, i.e., `logs-endpoint.events.file-*`, `logs-endpoint.events.network-*` (depending which one(s) you have), `ml_network_ded.all`, and copy the same in the **Name** field.
+    1. Enter the name of your respective index patterns in the **Index pattern** box, i.e., `logs-endpoint.events.file-*`, `logs-endpoint.events.network-*` (depending which one(s) you have), `ml_network_ded_ea.all`, and copy the same in the **Name** field.
     1. Select `@timestamp` under the **Timestamp** field and click on **Save data view to Kibana**.
-    1. Use the new data view (`logs-endpoint.events.network-*`, `logs-endpoint.events.file-*`, `ml_network_ded.all`) to create anomaly detection jobs for this package.
+    1. Use the new data view (`logs-endpoint.events.network-*`, `logs-endpoint.events.file-*`, `ml_network_ded_ea.all`) to create anomaly detection jobs for this package.
 1. **Add preconfigured anomaly detection jobs**: In **Stack Management -> Anomaly Detection Jobs**, you will see **Select data view or saved search**. Select the data view created in the previous step. Then under `Use preconfigured jobs` you will see **Data Exfiltration Detection**. If you do not see this card, events must be ingested from a source that matches the query specified in the [ded-ml file](https://github.com/elastic/integrations/blob/main/packages/ded/kibana/ml_module/ded-ml.json#L10), such as Elastic Defend. When you select the card, you will see pre-configured anomaly detection jobs that you can create depending on what makes the most sense for your environment. If you are using Elastic Defend to collect events, file events are in `logs-endpoint.events.file-*` and network events in `logs-endpoint.events.network-*`. If you are only collecting file or network events, select only the relevant jobs at this step.
 1. **Data view configuration for Dashboards**: For the dashboard to work as expected, the following settings need to be configured in Kibana. 
     1. You have started the above anomaly detection jobs.
     1. You have **read** access to `.ml-anomalies-shared` index or are assigned the `machine_learning_user` role. For more information on roles, please refer to [Built-in roles in Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html). Please be aware that a user who has access to the underlying machine learning results indices can see the results of _all_ jobs in _all_ spaces. Be mindful of granting permissions if you use Kibana spaces to control which users can see which machine learning results. For more information on machine learning privileges, refer to [setup-privileges](https://www.elastic.co/guide/en/machine-learning/current/setup.html#setup-privileges).
     1. After enabling the jobs, go to **Management > Stack Management > Kibana > Data Views**.  Click on **Create data view** with the following settings:
         - Name: `.ml-anomalies-shared`
-        - Index pattern : `.ml-anomalies-shared`
+        - Index pattern : `.ml-anomalies-shared*`
         - Select **Show Advanced settings** enable **Allow hidden and system indices**
         - Custom data view ID: `.ml-anomalies-shared`
 
@@ -42,18 +42,18 @@
 
 | Transform name      | Purpose                                     | Source index | Destination index        | Alias              |
 | ------------------- | ------------------------------------------- | ------------ | ------------------------ | ------------------ |
-| ded.pivot_transform | Collects network logs from your environment | logs-*       | ml_network_ded-[version] | ml_network_ded.all |
+| ded.pivot_transform_ea | Collects network logs from your environment | logs-*       | ml_network_ded_ea-[version] | ml_network_ded_ea.all |
 
 **Note**: The transform applies only to network data and does not currently support macOS network logs.
 
-When querying the destination index (`ml_network_ded-<VERSION>`) for network logs, we advise using the alias for the destination index (`ml_network_ded.all`). In the event that the underlying package is upgraded, the alias will aid in maintaining the previous findings. 
+When querying the destination index (`ml_network_ded_ea-<VERSION>`) for network logs, we advise using the alias for the destination index (`ml_network_ded_ea.all`). In the event that the underlying package is upgraded, the alias will aid in maintaining the previous findings. 
 
 ## Customize Data Exfiltration Detection Transform
 
 To customize filters in the Data Exfiltration Detection transform, follow the below steps. You can use these instructions to update basic settings or to update filters for fields such as `process.name`, `source.ip`, `destination.ip`, and others.
 1. To update settings such as retention policy, frequency, or destination configuration, stop the transform, click **Edit** from the **Actions** bar, make the required changes, and start the transform again.
 ![Data Exfiltration Detection transform](../img/ded_transform_update.png)
-1. To update the query filters, go to **Stack Management > Data > Transforms > `logs-ded.pivot_transform-default-<FLEET-TRANSFORM-VERSION>`**.
+1. To update the query filters, go to **Stack Management > Data > Transforms > `logs-ded.pivot_transform_ea-default-<FLEET-TRANSFORM-VERSION>`**.
 1. Click on the **Actions** bar at the far right of the transform and select the **Clone** option.
 ![Data Exfiltration Detection transform](../img/ded_transform_1.png)
 1. In the new **Clone transform** window, go to the **Search filter** and update any field values you want to add or remove. Click on the **Apply changes** button on the right side to save these changes. **Note:** The image below shows an example of filtering a new `process.name` as `explorer.exe`. You can follow a similar example and update the field value list based on your environment to help reduce noise and potential false positives.
@@ -71,13 +71,13 @@
 
 | Job                                                  | Description                                                                                                        | Supported Platform | Event Category |
 | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------ | ----- |
-| ded_high_sent_bytes_destination_geo_country_iso_code | Detects data exfiltration to an unusual geo-location (by country iso code).                                        | Linux, Windows     | network |
-| ded_high_sent_bytes_destination_ip                   | Detects data exfiltration to an unusual geo-location (by IP address).                                              | Linux, Windows     | network |
-| ded_high_sent_bytes_destination_port                 | Detects data exfiltration to an unusual destination port.                                                          | Linux, Windows     | network |
-| ded_high_sent_bytes_destination_region_name          | Detects data exfiltration to an unusual geo-location (by region name).                                             | Linux, Windows     | network |
-| ded_high_bytes_written_to_external_device            | Detects data exfiltration activity by identifying high bytes written to an external device.                        | Windows            | file |
-| ded_rare_process_writing_to_external_device          | Detects data exfiltration activity by identifying a writing event started by a rare process to an external device. | Windows            | file |
-| ded_high_bytes_written_to_external_device_airdrop    | Detects data exfiltration activity by identifying high bytes written to an external device via Airdrop.            | macOS              | file |
+| ded_high_sent_bytes_destination_geo_country_iso_code_ea | Detects data exfiltration to an unusual geo-location (by country iso code).                                        | Linux, Windows     | network |
+| ded_high_sent_bytes_destination_ip_ea                   | Detects data exfiltration to an unusual geo-location (by IP address).                                              | Linux, Windows     | network |
+| ded_high_sent_bytes_destination_port_ea                 | Detects data exfiltration to an unusual destination port.                                                          | Linux, Windows     | network |
+| ded_high_sent_bytes_destination_region_name_ea          | Detects data exfiltration to an unusual geo-location (by region name).                                             | Linux, Windows     | network |
+| ded_high_bytes_written_to_external_device_ea            | Detects data exfiltration activity by identifying high bytes written to an external device.                        | Windows            | file |
+| ded_rare_process_writing_to_external_device_ea          | Detects data exfiltration activity by identifying a writing event started by a rare process to an external device. | Windows            | file |
+| ded_high_bytes_written_to_external_device_airdrop_ea    | Detects data exfiltration activity by identifying high bytes written to an external device via Airdrop.            | macOS              | file |
 
 ## Customize ML jobs for Data Exfiltration Detection 
 
@@ -96,6 +96,28 @@
 ![Data Exfiltration Detection jobs](../img/ded_ml_job_6.png)
 1. Finally, assign a new Job ID, and click on **Create job**, and start the datafeed to apply the updated settings.
 
+## v3.0.0 and beyond
+
+v3.0.0 of the package introduces support for Entity Analytics (EA) in Elastic Stack version 9.4, adding new fields for proper entity resolution.
+
+- The new ML jobs include an `_ea` suffix in their names, as outlined below. New transforms and detection rules are also included.
+- Previously installed ML jobs, transforms, and rules will continue to run, allowing time to transition to the new Entity Analytics assets.
+- We recommend installing the new ML jobs and transforms first and verifying that they are properly set up, collecting data, and generating anomalies before upgrading to the latest detection rules included in 9.4.
+- The new Entity Analytics transforms write to separate destination indices postfixed with `_ea`. Create a new data view for the Entity Analytics anomaly detection jobs using the new destination indices/aliases listed below. Do not mix old and new transform destination indices in the same data view.
+- New dashboards are available in this version with the suffix "(Entity Analytics)" in the title. If you are still running jobs or transforms from before this version, the original dashboards without the suffix remain available.
+
+The new Entity Analytics ML job IDs are:
+- `ded_high_sent_bytes_destination_geo_country_iso_code_ea`
+- `ded_high_sent_bytes_destination_ip_ea`
+- `ded_high_sent_bytes_destination_port_ea`
+- `ded_high_sent_bytes_destination_region_name_ea`
+- `ded_high_bytes_written_to_external_device_ea`
+- `ded_rare_process_writing_to_external_device_ea`
+- `ded_high_bytes_written_to_external_device_airdrop_ea`
+
+The new Entity Analytics transforms are:
+- `ded.pivot_transform_ea` → destination index: `ml_network_ded_ea-3.0.0`, alias: `ml_network_ded_ea.latest`, `ml_network_ded_ea.all`
+
 ## v2.0.0 and beyond
 
 v2.0.0 of the package introduces breaking changes, namely deprecating detection rules from the package. To continue receiving updates to Data Exfiltration Detection, we recommend upgrading to v2.0.0 after doing the following:

packages/ded/elasticsearch/transform/pivot_transform/fields/fields.yml → packages/ded/elasticsearch/transform/pivot_transform_ea/fields/fields.yml

@@ -1,7 +1,13 @@
 - external: ecs
   name: host.name
+- external: ecs
+  name: host.id
 - external: ecs
   name: user.name
+- external: ecs
+  name: user.id
+- external: ecs
+  name: event.module
 - external: ecs
   name: event.category
 - external: ecs
@@ -27,4 +33,4 @@
 - external: ecs
   name: destination.geo.region_name
 - external: ecs
-  name: destination.geo.city_name
+  name: destination.geo.city_name

packages/ded/elasticsearch/transform/pivot_transform/transform.yml → packages/ded/elasticsearch/transform/pivot_transform_ea/transform.yml

@@ -1,12 +1,12 @@
 
 dest:
-  index: ml_network_ded-2.4.1
+  index: ml_network_ded_ea-3.0.0
   aliases:
-    - alias: ml_network_ded.latest
+    - alias: ml_network_ded_ea.latest
       move_on_creation: true
-    - alias: ml_network_ded.all
+    - alias: ml_network_ded_ea.all
       move_on_creation: false
-  pipeline: 2.4.1-ml_ded_ingest_pipeline
+  pipeline: 3.0.0-ml_ded_ingest_pipeline
 description: This transform runs every 30 minutes and collects network logs to detect data exfiltration in your environment for the past month up to the runtime.
 frequency: 30m
 pivot:
@@ -21,9 +21,18 @@ pivot:
     'host.name':
       terms:
         field: host.name
+    host.id:
+      terms:
+        field: host.id
     'user.name':
       terms:
         field: user.name
+    user.id:
+      terms:
+        field: user.id
+    event.module:
+      terms:
+        field: event.module
     'network.direction':
       terms:
         field: network.direction
@@ -94,5 +103,5 @@ sync:
     delay: 120s
     field: "@timestamp"
 _meta:
-  fleet_transform_version: 2.4.1
+  fleet_transform_version: 3.0.0
   run_as_kibana_system: false