[DE][9.4 & Serverless] New space id filter available in Kibana advanced settings for all detection rules

solutions/security/get-started/configure-advanced-settings.md

@@ -17,6 +17,7 @@ navigation_title: Configure advanced settings
 The advanced settings control the behavior of the {{security-app}}, such as:
 
 * Which indices {{elastic-sec}} uses to retrieve data
+* Which data stream namespaces detection rules search (when limited by namespace)
 * {{ml-cap}} anomaly score display threshold
 * The navigation menu style used throughout the {{security-app}}
 * Whether the news feed is displayed on the [Overview dashboard](/solutions/security/dashboards/overview-dashboard.md)
@@ -248,6 +249,22 @@ Even when the `excludedDataTiersForRuleExecution` advanced setting is enabled, i
 ::::
 
 
+## Limit detection rules to specific data stream namespaces [included-data-stream-namespaces-rule-execution]
+
+```yaml {applies_to}
+stack: ga 9.4
+serverless: ga
+```
+
+When configured, the **Include data stream namespaces in rule execution** setting (`securitySolution:includedDataStreamNamespacesForRuleExecution`) restricts which documents detection rules search. Only events whose `data_stream.namespace` field matches one of the specified namespaces are queried. This applies to all detection rules in the {{kib}} space and acts like a global filter on `data_stream.namespace`.
+
+Specify an array of namespace strings (for example, `namespace1`, `namespace2`). You can configure up to 50 namespaces. Leave the setting empty to search all namespaces (default behavior).
+
+::::{tip}
+If rules are not creating expected alerts or are missing data, check that this advanced setting is not filtering out the namespaces where your data is stored. Refer to [Troubleshoot missing alerts](../../../troubleshoot/security/detection-rules.md#troubleshoot-namespace-filter) for more information.
+::::
+
+
 ## Access privileged user monitoring
 ```yaml {applies_to}
 stack: removed 9.3, ga 9.1

troubleshoot/security/detection-rules.md

@@ -158,52 +158,47 @@
 
 ## Rule executions
 
-:::::{dropdown} Troubleshoot missing alerts
-:name: troubleshoot-signals
 
-When a rule fails to run close to its scheduled time, some alerts may be missing. There are a number of ways to try to resolve this issue:
+### Troubleshoot missing alerts [troubleshoot-signals]
 
-* [Troubleshoot gaps](#troubleshoot-gaps)
-* [Troubleshoot ingestion pipeline delay](#troubleshoot-ingestion-pipeline-delay)
-* [Troubleshoot missing alerts for {{ml}} jobs](#ml-job-compatibility)
-
-You can also use Task Manager in {{kib}} to troubleshoot background tasks and processes that may be related to missing alerts:
-
-* [Task Manager health monitoring](../../deploy-manage/monitor/kibana-task-manager-health-monitoring.md)
-* [Task Manager troubleshooting](../../troubleshoot/kibana/task-manager.md)
 
+When a rule fails to run close to its scheduled time, some alerts may be missing. There are a number of ways to try to resolve this issue.
 
-### Troubleshoot maximum alerts warning [troubleshoot-max-alerts]
+::::{dropdown} Troubleshoot maximum alerts warning
+:name: troubleshoot-max-alerts
 
 When a rule reaches the maximum number of alerts it can generate during a single rule execution, the following warning appears on the rule’s details page and in the rule execution log: `This rule reached the maximum alert limit for the rule execution. Some alerts were not created.`
 
 If you receive this warning, go to the rule’s **Alerts** tab and check for anything unexpected. Unexpected alerts might be created from data source issues or queries that are too broadly scoped. To further reduce alert volume, you can also add [rule exceptions](../../solutions/security/detect-and-alert/add-manage-exceptions.md) or [suppress alerts](../../solutions/security/detect-and-alert/alert-suppression.md).
 
+::::
+
 
-### Troubleshoot gaps [troubleshoot-gaps]
+::::{dropdown} Troubleshoot gaps
+:name: troubleshoot-gaps
 
 If you see values in the Gaps column in the Rule Monitoring table or on the Rule details page for a small number of rules, you can edit those rules and increase their additional look-back time.
 
 It’s recommended to set the `Additional look-back time` to at least 1 minute. This ensures there are no missing alerts when a rule doesn’t run exactly at its scheduled time.
 
 {{elastic-sec}} prevents duplication. Any duplicate alerts that are discovered during the `Additional look-back time` are *not* created.
 
-::::{note}
 If the rule that experiences gaps is an indicator match rule, see [how to tune indicator match rules](../../solutions/security/detect-and-alert/tune-detection-rules.md#tune-indicator-rules). Also note that {{elastic-sec}} provides [limited support for indicator match rules](../../solutions/security/detect-and-alert/indicator-match.md).
-::::
-
 
 If you see gaps for numerous rules:
 
 * If you restarted {{kib}} when many rules were activated, try deactivating them and then reactivating them in small batches at staggered intervals. This ensures {{kib}} does not attempt to run all the rules at the same time.
 * Consider adding another {{kib}} instance to your environment.
 
+::::
+
 
-### Troubleshoot ingestion pipeline delay [troubleshoot-ingestion-pipeline-delay]
+::::{dropdown} Troubleshoot ingestion pipeline delay
+:name: troubleshoot-ingestion-pipeline-delay
 
 Even if your rule runs at its scheduled time, there might still be missing alerts if your ingestion pipeline delay is greater than your rule interval + additional look-back time. Prebuilt rules have a minimum interval + additional look-back time of 6 minutes in {{stack}} version >=7.11.0. To avoid missed alerts for prebuilt rules, use caution to ensure that ingestion pipeline delays remain below 6 minutes.
 
 In addition, use caution when creating custom rule schedules to ensure that the specified interval + additional look-back time is greater than your deployment’s ingestion pipeline delay.
 
 You can reduce the number of missed alerts due to ingestion pipeline delay by specifying the `Timestamp override` field value to `event.ingested` in [advanced settings](../../solutions/security/detect-and-alert/common-rule-settings.md#rule-ui-advanced-params) during rule creation or editing. The detection engine uses the value from the `event.ingested` field as the timestamp when executing the rule.
 
@@ -214,24 +209,39 @@
 :screenshot:
 :::
 
+::::
+
+
+::::{dropdown} Troubleshoot namespace filter
+:name: troubleshoot-namespace-filter
 
-### Troubleshoot missing alerts for {{ml}} jobs [ml-job-compatibility]
 ```yaml {applies_to}
-stack: all
+stack: ga 9.4
+serverless: ga
 ```
 
+If detection rules are not creating expected alerts or appear to be missing data, the **Include data stream namespaces in rule execution** [advanced setting](../../solutions/security/get-started/configure-advanced-settings.md#included-data-stream-namespaces-rule-execution) may be limiting which documents are searched. When configured, only events with a matching `data_stream.namespace` value are queried by all rules in the {{kib}} space.
 
-{{ml-cap}} detection rules use {{ml}} jobs that have dependencies on data fields populated by the {{beats}} and {{agent}} integrations. In {{stack}} version 8.3, new {{ml}} jobs (prefixed with `v3`) were released to operate on the ECS fields available at that time.
+To verify:
 
-If you’re using 8.2 or earlier versions of {{beats}} or {{agent}} with {{stack}} version 8.3 or later, you may need to duplicate prebuilt rules or create new custom rules *before* you update the Elastic prebuilt rules. Once you update the prebuilt rules, they will only use `v3` {{ml}} jobs. Duplicating the relevant prebuilt rules before updating them ensures continued coverage by allowing you to keep using `v1` or `v2` jobs (in the duplicated rules) while also running the new `v3` jobs (in the updated prebuilt rules).
-
-::::{important}
-* Duplicated rules may result in duplicate anomaly detections and alerts.
-* Ensure that the relevant `v3` {{ml}} jobs are running before you update the Elastic prebuilt rules.
+1. Go to **{{stack-manage-app}}** → **Advanced Settings** (or **Project Settings** → **{{stack-manage-app}}** → **Advanced Settings** in {{serverless-short}}).
+2. Scroll to **Security Solution** and find **Include data stream namespaces in rule execution**.
+3. If the setting lists one or more namespaces, ensure your data is written to those namespaces (for example, check your {{fleet}} integration or data shipper configuration for the `data_stream.namespace` value). If your data uses different namespace values, add them to the setting or clear the setting to search all namespaces.
 
 ::::
 
 
+::::{dropdown} Troubleshoot missing alerts for {{ml}} jobs
+:name: ml-job-compatibility
+
+```yaml {applies_to}
+stack: all
+```
+
+{{ml-cap}} detection rules use {{ml}} jobs that have dependencies on data fields populated by the {{beats}} and {{agent}} integrations. In {{stack}} version 8.3, new {{ml}} jobs (prefixed with `v3`) were released to operate on the ECS fields available at that time.
+
+If you’re using 8.2 or earlier versions of {{beats}} or {{agent}} with {{stack}} version 8.3 or later, you may need to duplicate prebuilt rules or create new custom rules *before* you update the Elastic prebuilt rules. Once you update the prebuilt rules, they will only use `v3` {{ml}} jobs. Duplicating the relevant prebuilt rules before updating them ensures continued coverage by allowing you to keep using `v1` or `v2` jobs (in the duplicated rules) while also running the new `v3` jobs (in the updated prebuilt rules). Keep in mind that duplicated rules may result in duplicate anomaly detections and alerts, and ensure that the relevant `v3` {{ml}} jobs are running before you update the Elastic prebuilt rules.
+
 * If you only have **8.3 or later versions of {{beats}} and {{agent}}**: You can download or update your prebuilt rules and use the latest `v3` {{ml}} jobs. No additional action is required.
 * If you only have **8.2 or earlier versions of {{beats}} or {{agent}}**, or **a mix of old and new versions**: To continue using the `v1` and `v2` {{ml}} jobs specified by pre-8.3 prebuilt detection rules, you must duplicate affected prebuilt rules *before* updating them to the latest rule versions. The duplicated rules can continue using the same `v1` and `v2` {{ml}} jobs, and the updated prebuilt {{ml}} rules will use the new `v3` {{ml}} jobs.
 * If you have **a non-Elastic data shipper that gathers ECS-compatible events**: You can use the latest `v3` {{ml}} jobs with no additional action required, as long as your data shipper uses the latest ECS specifications. However, if you’re migrating from {{ml}} rules using `v1`/`v2` jobs, ensure that you start the relevant `v3` jobs before updating the Elastic prebuilt rules.
@@ -254,4 +264,9 @@
 * [Unusual Windows Process Calling the Metadata Service](detection-rules://rules/ml/credential_access_ml_windows_anomalous_metadata_process.md): `v3_windows_rare_metadata_process`
 * [Unusual Windows User Calling the Metadata Service](detection-rules://rules/ml/credential_access_ml_windows_anomalous_metadata_user.md): `v3_windows_rare_metadata_user`
 
-:::::
+::::
+
+You can also use Task Manager in {{kib}} to troubleshoot background tasks and processes that may be related to missing alerts:
+
+* [Task Manager health monitoring](../../deploy-manage/monitor/kibana-task-manager-health-monitoring.md)
+* [Task Manager troubleshooting](../../troubleshoot/kibana/task-manager.md)