From f9c2d6d36170175eb28be23cb319e4d1b1b25045 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 23 Oct 2024 13:34:34 -0700 Subject: [PATCH 01/51] 8.16 Elastic Security Release Notes --- docs/release-notes/8.16.asciidoc | 63 ++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 docs/release-notes/8.16.asciidoc diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc new file mode 100644 index 0000000000..bd732fd266 --- /dev/null +++ b/docs/release-notes/8.16.asciidoc @@ -0,0 +1,63 @@ +[[release-notes-header-8.16.0]] +== 8.16 + +[discrete] +[[release-notes-8.16.0]] +=== 8.16.0 + +[discrete] +[[known-issue-8.16.0]] +==== Known issues + + +[discrete] +[[breaking-changes-8.16.0]] +==== Breaking changes + +* During shutdown, Kibana now waits for all the ongoing requests to complete according to the `server.shutdownTimeout` setting. During that period, the incoming socket is closed and any new incoming requests are rejected. Before this update, new incoming requests received a response with the status code 503 and body { "message": "Kibana is shutting down and not accepting new incoming requests" }. + +[discrete] +[[features-8.16.0]] +==== New features + +// * Introduces a new API route for listing Entity Store entities: `GET /api/entity_store/entities/list` ({kibana-pull}192806[#192806]). +* Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). +* Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). +* Creates an option for {esql} and EQL rule previews `Show {es} requests, ran during rule executions`. When enabled, {es} queries that run during rule execution appear under "Preview logged requests" in the rule preview ({kibana-pull}191107[#191107]). + +* Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). +* Creates a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, a "Visualizations" section with Analyzer and Session view previews appears in the alerts and events flyouts ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531]). +* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data tiers from search during rule execution (does not apply to {esql} or Machine learning rules) ({kibana-pull}186908[#186908]). + +* Adds an "Insights" section containing any available misconfiguration and vulnerabilities findings to the alert and event flyouts ({kibana-pull}195509[#195509]). +* Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) +* Adds an {elastic-defend} integration policy advanced setting that, when enabled, adds file hashes to file events ({kibana-pull}192037[#192037]). + + +* Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). + +* Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). + +* cspm agentless +({kibana-pull}191557[#191557]). + +[discrete] +[[enhancements-8.16.0]] +==== Enhancements +* Enables you to resize expanded flyouts ({kibana-pull}192906[#192906]). +* Enables a settings menu for flyouts that lets you select between `Overlay` and `Push` display modes ({kibana-pull}182615[#182615]). +* Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). +* Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). +* Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843]). + +* Removed Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). + +* Updates the Get started tour for {elastic-sec} ({kibana-pull}192247[#192247]). + +* Speeds up loading times for various pages in {kib} ({kibana-pull}194241[#194241]). + +* Adds a kibana advanced setting `securitySolution:maxUnassociatedNotes` that allows you to limit the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). + +[discrete] +[[bug-fixes-8.16.0]] +==== Bug fixes From 9e8664bdcf450a4ca6d29fa84459b637ae1cf75b Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 23 Oct 2024 16:24:50 -0700 Subject: [PATCH 02/51] Adds 8.16 rns to index file --- docs/release-notes.asciidoc | 1 + docs/release-notes/8.16.asciidoc | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/release-notes.asciidoc b/docs/release-notes.asciidoc index 7a6c8db425..b9b5bc1fad 100644 --- a/docs/release-notes.asciidoc +++ b/docs/release-notes.asciidoc @@ -3,6 +3,7 @@ This section summarizes the changes in each release. +* <> * <> * <> * <> diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index bd732fd266..0ed1445858 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -51,12 +51,15 @@ * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843]). * Removed Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). +* Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480]). -* Updates the Get started tour for {elastic-sec} ({kibana-pull}192247[#192247]). +* Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Speeds up loading times for various pages in {kib} ({kibana-pull}194241[#194241]). -* Adds a kibana advanced setting `securitySolution:maxUnassociatedNotes` that allows you to limit the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). + + +* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes` that allows you to set the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). [discrete] [[bug-fixes-8.16.0]] From cda2bdd4fab118011d155424b2e35444eec89da7 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 23 Oct 2024 17:45:54 -0700 Subject: [PATCH 03/51] Completes first draft --- docs/release-notes/8.16.asciidoc | 43 ++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 13 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 0ed1445858..5931ff69ee 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -24,43 +24,60 @@ * Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). * Creates an option for {esql} and EQL rule previews `Show {es} requests, ran during rule executions`. When enabled, {es} queries that run during rule execution appear under "Preview logged requests" in the rule preview ({kibana-pull}191107[#191107]). - * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). * Creates a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, a "Visualizations" section with Analyzer and Session view previews appears in the alerts and events flyouts ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data tiers from search during rule execution (does not apply to {esql} or Machine learning rules) ({kibana-pull}186908[#186908]). - * Adds an "Insights" section containing any available misconfiguration and vulnerabilities findings to the alert and event flyouts ({kibana-pull}195509[#195509]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) * Adds an {elastic-defend} integration policy advanced setting that, when enabled, adds file hashes to file events ({kibana-pull}192037[#192037]). - - * Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). - +* Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). - -* cspm agentless -({kibana-pull}191557[#191557]). +* Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). [discrete] [[enhancements-8.16.0]] ==== Enhancements +* Enables you to open the Rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). * Enables you to resize expanded flyouts ({kibana-pull}192906[#192906]). * Enables a settings menu for flyouts that lets you select between `Overlay` and `Push` display modes ({kibana-pull}182615[#182615]). * Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). -* Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843]). +* Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). * Removed Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). -* Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480]). - +* Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). +* Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]) +* Improves Attack Discovery ({kibana-pull}195669[#195669]): +** AD can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} +** AD now combines related discoveries that would previously have appeared separately +** AD now detects and displays an error instead of hallucinated output + +* Adds an `Install and enable` button to the preuilt rule UI, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). +* Improves the `/upgrade/_perform` API endpoint used for upgrading rules to better handle Elastic rules that you customized, better handle special fields, and use `PUT` instead of `PATCH` logic for rule updates ({kibana-pull}191439[#191439]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). +* Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). +* Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). +* Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint Details flyout ({kibana-pull}184125[#184125]). +* Allows you to set the risk scoring engine to run automatically after you upload asset criticality data ({kibana-pull}187577[#187577]). -* Speeds up loading times for various pages in {kib} ({kibana-pull}194241[#194241]). +* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes` that allows you to set the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). +* Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). +* Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). +* Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). -* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes` that allows you to set the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). [discrete] [[bug-fixes-8.16.0]] ==== Bug fixes + +* Prevents an empty warning message from appearing for rule executions ({kibana-pull}186096[#186096]). +* Fixes an error that could occur during rule execution when the source index had a text field that was noncompliant with ECS ({kibana-pull}187673[#187673]). +* Removes unnecessary empty space below the Open Timeline modal's title ({kibana-pull}188837[#188837]). +* Added a tag that was missing from an FTR suite ({kibana-pull}189661[#189661]). +* Improves the Alerts table's performance ({kibana-pull}192827[#192827]). +* Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). +* Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). +* \ No newline at end of file From 5bedce81c7c3eb4d5c63678af2b0be7b9bb4d0ab Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Thu, 24 Oct 2024 10:13:49 -0400 Subject: [PATCH 04/51] Including 8.16 rn file --- docs/release-notes.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/release-notes.asciidoc b/docs/release-notes.asciidoc index b9b5bc1fad..8163df998e 100644 --- a/docs/release-notes.asciidoc +++ b/docs/release-notes.asciidoc @@ -66,6 +66,7 @@ This section summarizes the changes in each release. * <> * <> +include::release-notes/8.16.asciidoc[] include::release-notes/8.15.asciidoc[] include::release-notes/8.14.asciidoc[] include::release-notes/8.13.asciidoc[] From f7ea2059d873146d8cc9ea07581c06ba38e2b2f7 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Thu, 24 Oct 2024 10:19:34 -0700 Subject: [PATCH 05/51] minor updates --- docs/release-notes/8.16.asciidoc | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 5931ff69ee..d7c266f88b 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -41,18 +41,16 @@ * Enables you to open the Rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). * Enables you to resize expanded flyouts ({kibana-pull}192906[#192906]). * Enables a settings menu for flyouts that lets you select between `Overlay` and `Push` display modes ({kibana-pull}182615[#182615]). -* Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). - -* Removed Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). +* Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). * Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). -* Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]) +* Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]). +* Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). * Improves Attack Discovery ({kibana-pull}195669[#195669]): ** AD can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} ** AD now combines related discoveries that would previously have appeared separately ** AD now detects and displays an error instead of hallucinated output - * Adds an `Install and enable` button to the preuilt rule UI, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). * Improves the `/upgrade/_perform` API endpoint used for upgrading rules to better handle Elastic rules that you customized, better handle special fields, and use `PUT` instead of `PATCH` logic for rule updates ({kibana-pull}191439[#191439]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). @@ -60,15 +58,12 @@ * Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). * Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint Details flyout ({kibana-pull}184125[#184125]). * Allows you to set the risk scoring engine to run automatically after you upload asset criticality data ({kibana-pull}187577[#187577]). - * Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes` that allows you to set the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). * Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). - - [discrete] [[bug-fixes-8.16.0]] ==== Bug fixes @@ -80,4 +75,3 @@ * Improves the Alerts table's performance ({kibana-pull}192827[#192827]). * Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). * Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). -* \ No newline at end of file From f1d21dc766830dceecf69f26764dd265948126a3 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 25 Oct 2024 13:59:55 -0400 Subject: [PATCH 06/51] First draft of Endpoint PRs --- docs/release-notes/8.16.asciidoc | 39 +++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index d7c266f88b..2a148044b5 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -34,6 +34,19 @@ * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). +//The following are Defend/Endpoint PRs. Still need to clean them up. +* Host field size reduction is now disabled by default. Host field size reduction can be enabled via and advanced config directive `[os].advanced.set_extended_host_information` +* To reduce CPU usage, I/O, and event sizes, users can now opt into aggregation of process events. Related process events occurring in rapid succession will be combined into fewer aggregate events. Use `advanced.events.aggregate_process` in advanced policy to enable it. +* See 15114. +//This ^ is for Endpoint PR 14937 +* Events. +* Users can now opt into the collection of SHA-256 file hashes in file events. These events come with several caveats, including: +** This can greatly increase Endpoint's CPU and I/O utilization, impacting system responsiveness. +** This can significantly delay event enrichment, leading to Behavioral Protection rules firing too late to effectively stop malicious behavior. +** This can cause event processing queues to overflow, leading to dropped events. +** Many file events will not contain hashes. Hash collection is best-effort and not guaranteed to be present in every event. Hashes are collected asynchronously, shortly after the file activity. Hashes may be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without [read sharing](https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files). +* Enables the use of dynamic topics for the kafka output. Refer to https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#topic-option-kafka for more information. +* Integrates a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. [discrete] [[enhancements-8.16.0]] @@ -62,7 +75,22 @@ * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). * Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). - +//The following are Defend/Endpoint PRs. Still need to clean them up. +* Improved `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. +* Adds additional fields to Defend API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. +* Add a new Defend API event for [`DeviceIoControl`](https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol) calls to support detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. +* Upon start Endpoint service will update security artifacts before arming itself with policy. +* Events. +//Not sure what this ^ is for +* Adds user password sock5 proxy authentication support embedded in URL: socks5:// : @. +* Improve error messages when Endpoint receives invalid or unsupported cryptographic keys via policy. +* If Defend loses connectivity to Agent for an extended period of time, it will notify Fleet that it is "orphaned." Fleet can use this information to provide additional context to the user. +* Adds SOCKS5 proxy support to Logstash output. +* On Windows, Defend will now use [Intel CET and AMD Shadow Stacks](https://www.elastic.co/security-labs/finding-truth-in-the-shadows) to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. +* Restore Defend support for Windows Server 2012, which was removed in 8.13.0. +* Improvements to Defend's caching to reduce memory usage on Windows. +* Reduced size of process events by default, reducing excessive process ancestry entries and shortening the entity id. +* Improve reliability and system resource usage of Defend's Windows network driver. [discrete] [[bug-fixes-8.16.0]] @@ -75,3 +103,12 @@ * Improves the Alerts table's performance ({kibana-pull}192827[#192827]). * Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). * Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). +//The following are Defend/Endpoint PRs. Still need to clean them up. +* Fixed a bug where network event deduplication logic could incorrectly drop Linux network events. +* Fixes a bug where Windows API events may be dropped if they contain Unicode characters that cannot be converted to ANSI. +* Track loopback (part 3). +* If Defend is unable to enrich a memory region in an API event, it will now remove that field instead of emitting an empty `memory_region`. +* Fix a bug where Defend can fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This may result in dropped or unattributed API events. +* When requested to use fqdn in the `host.name` field, endpoint will now report the fqdn exactly as the os reports it, rather than lowercasing by default. This will ensure host name uniformity with beats products. +* Fixes a bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. +* Fixed an issue where a busy Kafka connection could lead to an Endpoint crash. \ No newline at end of file From a6f17d2e65a76e2e89e223ac0d7366d4c79ac52f Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 25 Oct 2024 18:33:54 -0400 Subject: [PATCH 07/51] First batch of endpoint revisions --- docs/release-notes/8.16.asciidoc | 68 ++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 29 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 2a148044b5..b1825def0e 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -9,6 +9,22 @@ [[known-issue-8.16.0]] ==== Known issues +// tag::known-issue-189676[] +[discrete] +.Tags appear in Elastic AI Assistant's responses +[%collapsible] +==== +*Details* + +On August 1, 2024, it was discovered that Elastic AI Assistant's responses when using Bedrock Sonnet 3.5 may include `` tags, for example `` ({kibana-issue}189676[#189676]). + +==== +// end::known-issue-189676[] + +[discrete] +[[deprecations-8.16.0]] +==== Deprecations +* Reduced the size of the host field for endpoint events. +//14677 - noting because this was missed by the rn tool. [discrete] [[breaking-changes-8.16.0]] @@ -35,18 +51,16 @@ * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). //The following are Defend/Endpoint PRs. Still need to clean them up. -* Host field size reduction is now disabled by default. Host field size reduction can be enabled via and advanced config directive `[os].advanced.set_extended_host_information` -* To reduce CPU usage, I/O, and event sizes, users can now opt into aggregation of process events. Related process events occurring in rapid succession will be combined into fewer aggregate events. Use `advanced.events.aggregate_process` in advanced policy to enable it. -* See 15114. -//This ^ is for Endpoint PR 14937 -* Events. -* Users can now opt into the collection of SHA-256 file hashes in file events. These events come with several caveats, including: -** This can greatly increase Endpoint's CPU and I/O utilization, impacting system responsiveness. -** This can significantly delay event enrichment, leading to Behavioral Protection rules firing too late to effectively stop malicious behavior. +* Host field size reduction is now turned off by default on the {elastic-defend} integration policy. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. +* To reduce CPU usage, I/O, and event sizes, users can now opt into process event aggregation when configuring an {elastic-defend} integration policy. Related process events that occur in rapid succession aer combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. +* To reduce CPU usage, I/O, and event sizes, users can now opt out of MD5, SHA-1, and/or SHA-256 hashes in events when configuring an {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1. +* Users can now opt into the collection of SHA-256 file hashes in file events when configuring an {elastic-defend} integration policy. If you choose to opt in, consider the following caveats: +** This can greatly increase {elastic-defend}'s CPU and I/O utilization, impacting system responsiveness. +** This can significantly delay event enrichment, leading to Behavioral Protection rules firing to late to effectively stop malicious behavior. ** This can cause event processing queues to overflow, leading to dropped events. -** Many file events will not contain hashes. Hash collection is best-effort and not guaranteed to be present in every event. Hashes are collected asynchronously, shortly after the file activity. Hashes may be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without [read sharing](https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files). -* Enables the use of dynamic topics for the kafka output. Refer to https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#topic-option-kafka for more information. -* Integrates a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. +** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, shortly after the file activity. Hashes may be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without (https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files)[read sharing]. +* Improves {elastic-defend} by supporting the use of dynamic topics for the kafka output. Refer to {filebeat-ref}/kafka-output.html#topic-option-kafka[topic] for more information. +* Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. [discrete] [[enhancements-8.16.0]] @@ -77,20 +91,17 @@ * Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). //The following are Defend/Endpoint PRs. Still need to clean them up. * Improved `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. -* Adds additional fields to Defend API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. -* Add a new Defend API event for [`DeviceIoControl`](https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol) calls to support detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. -* Upon start Endpoint service will update security artifacts before arming itself with policy. -* Events. -//Not sure what this ^ is for -* Adds user password sock5 proxy authentication support embedded in URL: socks5:// : @. -* Improve error messages when Endpoint receives invalid or unsupported cryptographic keys via policy. -* If Defend loses connectivity to Agent for an extended period of time, it will notify Fleet that it is "orphaned." Fleet can use this information to provide additional context to the user. -* Adds SOCKS5 proxy support to Logstash output. -* On Windows, Defend will now use [Intel CET and AMD Shadow Stacks](https://www.elastic.co/security-labs/finding-truth-in-the-shadows) to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. -* Restore Defend support for Windows Server 2012, which was removed in 8.13.0. -* Improvements to Defend's caching to reduce memory usage on Windows. +* Adds additional fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. +* Add a new {elastic-defend} API event for (https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol)[`DeviceIoControl`] calls to support detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. +* Upon start, Endpoint service will update security artifacts before arming itself with policy. +* Improves error messages when {elastic-defend} receives invalid or unsupported cryptographic keys via policy. +* If {elastic-defend} loses connectivity to {agent} for an extended period of time, it will notify {fleet} that it is "orphaned." {fleet} can use this information to provide additional context to the user. +* Adds SOCKS5 proxy support to {ls} output. +* On Windows, Defend will now use (https://www.elastic.co/security-labs/finding-truth-in-the-shadows)[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. +* Restore {elastic-defend} support for Windows Server 2012, which was removed in 8.13.0. +* Improvements to {elastic-defend}'s caching to reduce memory usage on Windows. * Reduced size of process events by default, reducing excessive process ancestry entries and shortening the entity id. -* Improve reliability and system resource usage of Defend's Windows network driver. +* Improve reliability and system resource usage of {elastic-defend}'s Windows network driver. [discrete] [[bug-fixes-8.16.0]] @@ -106,9 +117,8 @@ //The following are Defend/Endpoint PRs. Still need to clean them up. * Fixed a bug where network event deduplication logic could incorrectly drop Linux network events. * Fixes a bug where Windows API events may be dropped if they contain Unicode characters that cannot be converted to ANSI. -* Track loopback (part 3). -* If Defend is unable to enrich a memory region in an API event, it will now remove that field instead of emitting an empty `memory_region`. -* Fix a bug where Defend can fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This may result in dropped or unattributed API events. -* When requested to use fqdn in the `host.name` field, endpoint will now report the fqdn exactly as the os reports it, rather than lowercasing by default. This will ensure host name uniformity with beats products. +* If {elastic-defend} is unable to enrich a memory region in an API event, it will now remove that field instead of emitting an empty `memory_region`. +* Fix a bug where {elastic-defend} can fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This may result in dropped or unattributed API events. +* When requested to use fqdn in the `host.name` field, endpoint will now report the fqdn exactly as the os reports it, rather than lowercasing by default. This will ensure host name uniformity with beats products. * Fixes a bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. -* Fixed an issue where a busy Kafka connection could lead to an Endpoint crash. \ No newline at end of file +* Fixed an issue where a busy Kafka connection could lead to {elastic-defend} crashing. \ No newline at end of file From b79e2036d902d64c0ca140c5ae852c15662c955d Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 25 Oct 2024 22:48:27 -0400 Subject: [PATCH 08/51] Second batch of edits for Endpoint PRs --- docs/release-notes/8.16.asciidoc | 57 +++++++++++++++----------------- 1 file changed, 27 insertions(+), 30 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index b1825def0e..87e2788a2c 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -50,16 +50,15 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). -//The following are Defend/Endpoint PRs. Still need to clean them up. * Host field size reduction is now turned off by default on the {elastic-defend} integration policy. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. -* To reduce CPU usage, I/O, and event sizes, users can now opt into process event aggregation when configuring an {elastic-defend} integration policy. Related process events that occur in rapid succession aer combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. -* To reduce CPU usage, I/O, and event sizes, users can now opt out of MD5, SHA-1, and/or SHA-256 hashes in events when configuring an {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1. -* Users can now opt into the collection of SHA-256 file hashes in file events when configuring an {elastic-defend} integration policy. If you choose to opt in, consider the following caveats: -** This can greatly increase {elastic-defend}'s CPU and I/O utilization, impacting system responsiveness. -** This can significantly delay event enrichment, leading to Behavioral Protection rules firing to late to effectively stop malicious behavior. -** This can cause event processing queues to overflow, leading to dropped events. -** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, shortly after the file activity. Hashes may be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without (https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files)[read sharing]. -* Improves {elastic-defend} by supporting the use of dynamic topics for the kafka output. Refer to {filebeat-ref}/kafka-output.html#topic-option-kafka[topic] for more information. +* To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. +* To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1. +* You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: +** This can greatly increase {elastic-defend}'s CPU and I/O utilization, and impact system responsiveness. +** This can significantly delay event enrichment, and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. +** This can cause event processing queues to overflow, and lead to dropped events. +** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without (https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files)[read sharing]. +* Improves {elastic-defend} enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the kafka output. * Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. [discrete] @@ -89,19 +88,18 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). * Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). -//The following are Defend/Endpoint PRs. Still need to clean them up. -* Improved `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. -* Adds additional fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. -* Add a new {elastic-defend} API event for (https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol)[`DeviceIoControl`] calls to support detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. -* Upon start, Endpoint service will update security artifacts before arming itself with policy. -* Improves error messages when {elastic-defend} receives invalid or unsupported cryptographic keys via policy. -* If {elastic-defend} loses connectivity to {agent} for an extended period of time, it will notify {fleet} that it is "orphaned." {fleet} can use this information to provide additional context to the user. -* Adds SOCKS5 proxy support to {ls} output. -* On Windows, Defend will now use (https://www.elastic.co/security-labs/finding-truth-in-the-shadows)[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. -* Restore {elastic-defend} support for Windows Server 2012, which was removed in 8.13.0. -* Improvements to {elastic-defend}'s caching to reduce memory usage on Windows. -* Reduced size of process events by default, reducing excessive process ancestry entries and shortening the entity id. -* Improve reliability and system resource usage of {elastic-defend}'s Windows network driver. +* Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. +* Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. +* Adds a new {elastic-defend} API event for (https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol)[`DeviceIoControl`] calls to support the detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. +* Ensures security artifacts are updated when the {elastic-defend} service starts. +* Improves error messages that are returned when {elastic-defend} receives invalid or unsupported cryptographic keys from the {elastic-defend} policy. +* Ensures that {elastic-defend} tells {fleet} that it's `orphaned` if the connection between {elastic-defend} and {agent} stops for an extended period of time. {fleet} uses this information to provide you with additional troubleshooting context. +* Adds SOCKS5 proxy support to {ls}'s output. +* Ensures that on Windows, {elastic-defend} uses (https://www.elastic.co/security-labs/finding-truth-in-the-shadows)[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. +* Restore {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0. +* Improves {elastic-defend}'s caching to reduce memory usage on Windows. +* Enhances {elastic-defend} by reducing the size of process events, which reduces excessive process ancestry entries and shortens the entity ID. +* Improves the reliability and system resource usage of {elastic-defend}'s Windows network driver. [discrete] [[bug-fixes-8.16.0]] @@ -114,11 +112,10 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Improves the Alerts table's performance ({kibana-pull}192827[#192827]). * Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). * Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). -//The following are Defend/Endpoint PRs. Still need to clean them up. -* Fixed a bug where network event deduplication logic could incorrectly drop Linux network events. -* Fixes a bug where Windows API events may be dropped if they contain Unicode characters that cannot be converted to ANSI. -* If {elastic-defend} is unable to enrich a memory region in an API event, it will now remove that field instead of emitting an empty `memory_region`. -* Fix a bug where {elastic-defend} can fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This may result in dropped or unattributed API events. -* When requested to use fqdn in the `host.name` field, endpoint will now report the fqdn exactly as the os reports it, rather than lowercasing by default. This will ensure host name uniformity with beats products. -* Fixes a bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. -* Fixed an issue where a busy Kafka connection could lead to {elastic-defend} crashing. \ No newline at end of file +* Fixes an {elastic-defend} bug where network event deduplication logic could incorrectly drop Linux network events. +* Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. +* Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields. +* Fixes an {elastic-defend} bug where {elastic-defend} doesn't properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. +* If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. This ensures host name uniformity with {beats} products. +* Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. +* Prevents {elastic-defend} from crashing if a Kafka connection is busy. \ No newline at end of file From 85af7dfa4f8239446e4c29ab981400ac1f0f6748 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Sat, 26 Oct 2024 18:42:43 -0400 Subject: [PATCH 09/51] Edits endpoint, DE, and TH rns --- docs/release-notes/8.16.asciidoc | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 87e2788a2c..e29ce7d872 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -39,10 +39,10 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when // * Introduces a new API route for listing Entity Store entities: `GET /api/entity_store/entities/list` ({kibana-pull}192806[#192806]). * Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). -* Creates an option for {esql} and EQL rule previews `Show {es} requests, ran during rule executions`. When enabled, {es} queries that run during rule execution appear under "Preview logged requests" in the rule preview ({kibana-pull}191107[#191107]). +* Provides a way to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). -* Creates a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, a "Visualizations" section with Analyzer and Session view previews appears in the alerts and events flyouts ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531]). -* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data tiers from search during rule execution (does not apply to {esql} or Machine learning rules) ({kibana-pull}186908[#186908]). +* Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531]). +* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). * Adds an "Insights" section containing any available misconfiguration and vulnerabilities findings to the alert and event flyouts ({kibana-pull}195509[#195509]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) * Adds an {elastic-defend} integration policy advanced setting that, when enabled, adds file hashes to file events ({kibana-pull}192037[#192037]). @@ -50,14 +50,14 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). -* Host field size reduction is now turned off by default on the {elastic-defend} integration policy. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. +* The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. * To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1. * You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: ** This can greatly increase {elastic-defend}'s CPU and I/O utilization, and impact system responsiveness. ** This can significantly delay event enrichment, and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. ** This can cause event processing queues to overflow, and lead to dropped events. -** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without (https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files)[read sharing]. +** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing]. * Improves {elastic-defend} enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the kafka output. * Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. @@ -65,8 +65,8 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when [[enhancements-8.16.0]] ==== Enhancements * Enables you to open the Rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). -* Enables you to resize expanded flyouts ({kibana-pull}192906[#192906]). -* Enables a settings menu for flyouts that lets you select between `Overlay` and `Push` display modes ({kibana-pull}182615[#182615]). +* Allows you to resize the alert and event details flyouts ({kibana-pull}192906[#192906]). +* Allows you display the alert and event details flyouts over the Alerts table or next to it ({kibana-pull}182615[#182615]). * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). * Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). @@ -90,12 +90,12 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). * Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. * Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. -* Adds a new {elastic-defend} API event for (https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol)[`DeviceIoControl`] calls to support the detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. +* Adds a new {elastic-defend} API event for https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol[`DeviceIoControl`] calls to support the detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. * Ensures security artifacts are updated when the {elastic-defend} service starts. * Improves error messages that are returned when {elastic-defend} receives invalid or unsupported cryptographic keys from the {elastic-defend} policy. * Ensures that {elastic-defend} tells {fleet} that it's `orphaned` if the connection between {elastic-defend} and {agent} stops for an extended period of time. {fleet} uses this information to provide you with additional troubleshooting context. * Adds SOCKS5 proxy support to {ls}'s output. -* Ensures that on Windows, {elastic-defend} uses (https://www.elastic.co/security-labs/finding-truth-in-the-shadows)[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. +* Ensures that on Windows, {elastic-defend} uses https://www.elastic.co/security-labs/finding-truth-in-the-shadows[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. * Restore {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0. * Improves {elastic-defend}'s caching to reduce memory usage on Windows. * Enhances {elastic-defend} by reducing the size of process events, which reduces excessive process ancestry entries and shortens the entity ID. @@ -106,8 +106,8 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when ==== Bug fixes * Prevents an empty warning message from appearing for rule executions ({kibana-pull}186096[#186096]). -* Fixes an error that could occur during rule execution when the source index had a text field that was noncompliant with ECS ({kibana-pull}187673[#187673]). -* Removes unnecessary empty space below the Open Timeline modal's title ({kibana-pull}188837[#188837]). +* Fixes an error that could occur during rule execution when the source index had a non-ECS-compliant text field ({kibana-pull}187673[#187673]). +* Removes unnecessary empty space below the title of the Open Timeline modal ({kibana-pull}188837[#188837]). * Added a tag that was missing from an FTR suite ({kibana-pull}189661[#189661]). * Improves the Alerts table's performance ({kibana-pull}192827[#192827]). * Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). @@ -116,6 +116,6 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. * Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields. * Fixes an {elastic-defend} bug where {elastic-defend} doesn't properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. -* If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. This ensures host name uniformity with {beats} products. +* Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. -* Prevents {elastic-defend} from crashing if a Kafka connection is busy. \ No newline at end of file +* Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. \ No newline at end of file From bf6eb321b747bfd4b342919fe6e16f3396060db4 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Wed, 30 Oct 2024 00:34:06 -0400 Subject: [PATCH 10/51] More minor edits --- docs/release-notes/8.16.asciidoc | 33 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index e29ce7d872..fd47e7899e 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -20,17 +20,11 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when ==== // end::known-issue-189676[] -[discrete] -[[deprecations-8.16.0]] -==== Deprecations -* Reduced the size of the host field for endpoint events. -//14677 - noting because this was missed by the rn tool. - [discrete] [[breaking-changes-8.16.0]] ==== Breaking changes -* During shutdown, Kibana now waits for all the ongoing requests to complete according to the `server.shutdownTimeout` setting. During that period, the incoming socket is closed and any new incoming requests are rejected. Before this update, new incoming requests received a response with the status code 503 and body { "message": "Kibana is shutting down and not accepting new incoming requests" }. +* During shutdown, {kib} now waits for all the ongoing requests to complete according to the `server.shutdownTimeout` setting. During that period, the incoming socket is closed and any new incoming requests are rejected. Before this update, new incoming requests received a response with the status code 503 and body { "message": "{kib} is shutting down and not accepting new incoming requests" }. [discrete] [[features-8.16.0]] @@ -43,9 +37,11 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). * Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). -* Adds an "Insights" section containing any available misconfiguration and vulnerabilities findings to the alert and event flyouts ({kibana-pull}195509[#195509]). +* Introduces the ability to add notes to alerts and events, in addition to Timeline. +//PR for notes feature is incoming. +* Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) -* Adds an {elastic-defend} integration policy advanced setting that, when enabled, adds file hashes to file events ({kibana-pull}192037[#192037]). +* Adds an {elastic-defend} integration policy advanced setting that, adds file hashes to file events when enabled ({kibana-pull}192037[#192037]). * Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). @@ -65,29 +61,28 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when [[enhancements-8.16.0]] ==== Enhancements * Enables you to open the Rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). -* Allows you to resize the alert and event details flyouts ({kibana-pull}192906[#192906]). -* Allows you display the alert and event details flyouts over the Alerts table or next to it ({kibana-pull}182615[#182615]). +* Allows you to resize the alert and event details flyouts and choose how it's displayed (over the Alerts table or next to it) ({kibana-pull}192906[#192906], {kibana-pull}182615[#182615]). * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). * Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). * Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). * Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]). * Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). -* Improves Attack Discovery ({kibana-pull}195669[#195669]): -** AD can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} -** AD now combines related discoveries that would previously have appeared separately -** AD now detects and displays an error instead of hallucinated output -* Adds an `Install and enable` button to the preuilt rule UI, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). +* Improves Attack Discovery in the following ways ({kibana-pull}195669[#195669]): +** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} +** Attack Discovery now combines related discoveries that would previously have appeared separately +** Attack Discovery now detects and displays an error instead of hallucinated output +* Adds an **Install and enable** button to the prebuilt rule UI, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). * Improves the `/upgrade/_perform` API endpoint used for upgrading rules to better handle Elastic rules that you customized, better handle special fields, and use `PUT` instead of `PATCH` logic for rule updates ({kibana-pull}191439[#191439]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). * Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). -* Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint Details flyout ({kibana-pull}184125[#184125]). +* Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint details flyout ({kibana-pull}184125[#184125]). * Allows you to set the risk scoring engine to run automatically after you upload asset criticality data ({kibana-pull}187577[#187577]). -* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes` that allows you to set the maximum number of notes that are not associated with a timeline ({kibana-pull}194947[#194947]). +* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). -* Improves network previews in the Alert details flyout ({kibana-pull}190560[#190560]). +* Improves network previews in the alert details flyout ({kibana-pull}190560[#190560]). * Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. * Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. * Adds a new {elastic-defend} API event for https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol[`DeviceIoControl`] calls to support the detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. From 0dbe3c6f38700a066da2d47c47132f28236450d4 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 30 Oct 2024 13:03:09 -0400 Subject: [PATCH 11/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index fd47e7899e..44f970cb93 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -40,7 +40,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Introduces the ability to add notes to alerts and events, in addition to Timeline. //PR for notes feature is incoming. * Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). -* Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) +* Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) * Adds an {elastic-defend} integration policy advanced setting that, adds file hashes to file events when enabled ({kibana-pull}192037[#192037]). * Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). From 70ffdf5cff6c85bae1eedb1a607f5d363fa2b89c Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 30 Oct 2024 13:03:21 -0400 Subject: [PATCH 12/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 44f970cb93..fd92d0bbfa 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -48,7 +48,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. -* To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1. +* To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. * You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: ** This can greatly increase {elastic-defend}'s CPU and I/O utilization, and impact system responsiveness. ** This can significantly delay event enrichment, and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. From fe263d2e8b8db5566305e353cd36c5f9bf825a7e Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 30 Oct 2024 13:03:28 -0400 Subject: [PATCH 13/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index fd92d0bbfa..3853abb8b4 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -78,7 +78,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). * Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). * Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint details flyout ({kibana-pull}184125[#184125]). -* Allows you to set the risk scoring engine to run automatically after you upload asset criticality data ({kibana-pull}187577[#187577]). +* Allows you to recalculate entity risk scores immediately after you upload asset criticality data ({kibana-pull}187577[#187577]). * Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). From 8ab9343e249b4b383a7606a3c1d17444a7152b60 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 1 Nov 2024 10:07:16 -0400 Subject: [PATCH 14/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Gabriel Landau <42078554+gabriellandau@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 3853abb8b4..24b9e6753d 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -81,7 +81,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to recalculate entity risk scores immediately after you upload asset criticality data ({kibana-pull}187577[#187577]). * Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). -* Allows you to disable the defend hardware call stacks feature ({kibana-pull}190553[#190553]). +* Allows you to disable the {elastic-defend} hardware call stacks feature ({kibana-pull}190553[#190553]). * Improves network previews in the alert details flyout ({kibana-pull}190560[#190560]). * Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. * Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. From 7af796bb0e1d100407e84a22d2faf5ccaebffc3c Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 1 Nov 2024 10:07:26 -0400 Subject: [PATCH 15/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Gabriel Landau <42078554+gabriellandau@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 24b9e6753d..5a4f6fc16b 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -89,7 +89,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Ensures security artifacts are updated when the {elastic-defend} service starts. * Improves error messages that are returned when {elastic-defend} receives invalid or unsupported cryptographic keys from the {elastic-defend} policy. * Ensures that {elastic-defend} tells {fleet} that it's `orphaned` if the connection between {elastic-defend} and {agent} stops for an extended period of time. {fleet} uses this information to provide you with additional troubleshooting context. -* Adds SOCKS5 proxy support to {ls}'s output. +* Adds SOCKS5 proxy support to {elastic-defend}'s {ls} output. * Ensures that on Windows, {elastic-defend} uses https://www.elastic.co/security-labs/finding-truth-in-the-shadows[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. * Restore {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0. * Improves {elastic-defend}'s caching to reduce memory usage on Windows. From 5d575c3f3f94da7303eaf97a8dc788a088708595 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Fri, 1 Nov 2024 08:38:00 -0700 Subject: [PATCH 16/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 5a4f6fc16b..b0aa5357d8 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -31,6 +31,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when ==== New features // * Introduces a new API route for listing Entity Store entities: `GET /api/entity_store/entities/list` ({kibana-pull}192806[#192806]). +* Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]). * Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). * Provides a way to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). From ddde9ebf2ea8812389236a1de5d35805b84948b9 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 10:13:26 -0500 Subject: [PATCH 17/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index b0aa5357d8..c464014091 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -74,7 +74,6 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when ** Attack Discovery now combines related discoveries that would previously have appeared separately ** Attack Discovery now detects and displays an error instead of hallucinated output * Adds an **Install and enable** button to the prebuilt rule UI, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). -* Improves the `/upgrade/_perform` API endpoint used for upgrading rules to better handle Elastic rules that you customized, better handle special fields, and use `PUT` instead of `PATCH` logic for rule updates ({kibana-pull}191439[#191439]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). * Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). From 8012fa081ff116975ff8bf2944399fea3cd88802 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 10:16:27 -0500 Subject: [PATCH 18/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Georgii Gorbachev --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index c464014091..54f31caee1 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -73,7 +73,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when ** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} ** Attack Discovery now combines related discoveries that would previously have appeared separately ** Attack Discovery now detects and displays an error instead of hallucinated output -* Adds an **Install and enable** button to the prebuilt rule UI, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). +* Adds an **Install and enable** button to the **Add Elastic Rules** page, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). * Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). From 9d8d0353a68c286a61b4468876f17db70155fd1d Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 14:09:13 -0500 Subject: [PATCH 19/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 54f31caee1..3af59108c6 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -42,7 +42,6 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when //PR for notes feature is incoming. * Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) -* Adds an {elastic-defend} integration policy advanced setting that, adds file hashes to file events when enabled ({kibana-pull}192037[#192037]). * Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). From c006928dc9a212958fd42e5986d28983678936ea Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 15:19:05 -0500 Subject: [PATCH 20/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 3af59108c6..55a2c69ba2 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -36,7 +36,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). * Provides a way to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). -* Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531]). +* Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). * Introduces the ability to add notes to alerts and events, in addition to Timeline. //PR for notes feature is incoming. From d1fd7fd1c89c00fa8a4b7077ee4abf16c3022766 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 15:19:40 -0500 Subject: [PATCH 21/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 55a2c69ba2..5cc51ed918 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -102,7 +102,6 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Prevents an empty warning message from appearing for rule executions ({kibana-pull}186096[#186096]). * Fixes an error that could occur during rule execution when the source index had a non-ECS-compliant text field ({kibana-pull}187673[#187673]). * Removes unnecessary empty space below the title of the Open Timeline modal ({kibana-pull}188837[#188837]). -* Added a tag that was missing from an FTR suite ({kibana-pull}189661[#189661]). * Improves the Alerts table's performance ({kibana-pull}192827[#192827]). * Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). * Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). From 320eff9cb9409b593064521678dca75c39dcace5 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Mon, 4 Nov 2024 16:21:35 -0800 Subject: [PATCH 22/51] Adds Automatic Import PRs --- docs/release-notes/8.16.asciidoc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 5cc51ed918..0b98946629 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -64,6 +64,8 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to resize the alert and event details flyouts and choose how it's displayed (over the Alerts table or next to it) ({kibana-pull}192906[#192906], {kibana-pull}182615[#182615]). * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). +* Allows Automatic Import to analyze a larger number of sample events when generating a new integration ({kibana-pull}196233[#196233]). +* Improves Automatic Import's ability to recognize events formatted as CSV ({kibana-pull}196228[#196228]). * Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). * Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). * Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]). @@ -111,4 +113,5 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Fixes an {elastic-defend} bug where {elastic-defend} doesn't properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. * Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. -* Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. \ No newline at end of file +* Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. +* Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). \ No newline at end of file From bc034e242ce3972b8089d5d08ff856c234b97388 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 19:30:24 -0500 Subject: [PATCH 23/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Gabriel Landau <42078554+gabriellandau@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 0b98946629..283555f4ab 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -82,7 +82,6 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to recalculate entity risk scores immediately after you upload asset criticality data ({kibana-pull}187577[#187577]). * Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). -* Allows you to disable the {elastic-defend} hardware call stacks feature ({kibana-pull}190553[#190553]). * Improves network previews in the alert details flyout ({kibana-pull}190560[#190560]). * Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. * Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. From 684548bf302dd66846419f275ba6089bbe2876aa Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 4 Nov 2024 19:35:41 -0500 Subject: [PATCH 24/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 283555f4ab..178b5eb9f9 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -90,7 +90,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Improves error messages that are returned when {elastic-defend} receives invalid or unsupported cryptographic keys from the {elastic-defend} policy. * Ensures that {elastic-defend} tells {fleet} that it's `orphaned` if the connection between {elastic-defend} and {agent} stops for an extended period of time. {fleet} uses this information to provide you with additional troubleshooting context. * Adds SOCKS5 proxy support to {elastic-defend}'s {ls} output. -* Ensures that on Windows, {elastic-defend} uses https://www.elastic.co/security-labs/finding-truth-in-the-shadows[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables detection of certain defense evasions. +* Ensures that on Windows, {elastic-defend} uses https://www.elastic.co/security-labs/finding-truth-in-the-shadows[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables the detection of certain defense evasions. You can turn this feature off in {elastic-defend} <> ({kibana-pull}190553[#190553]). * Restore {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0. * Improves {elastic-defend}'s caching to reduce memory usage on Windows. * Enhances {elastic-defend} by reducing the size of process events, which reduces excessive process ancestry entries and shortens the entity ID. From 6b9f91885a103d375498f6e393c62aa88d047bc5 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Tue, 5 Nov 2024 13:45:23 -0500 Subject: [PATCH 25/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 178b5eb9f9..892cc49dbb 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -109,7 +109,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Fixes an {elastic-defend} bug where network event deduplication logic could incorrectly drop Linux network events. * Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. * Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields. -* Fixes an {elastic-defend} bug where {elastic-defend} doesn't properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. +* Fixes a bug where {elastic-defend} could fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. * Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. * Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. From 0a3b585c1f2203bdfd77ccb31859b61b7d96b809 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 5 Nov 2024 15:15:59 -0500 Subject: [PATCH 26/51] jatin's feedback --- docs/release-notes/8.16.asciidoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 892cc49dbb..e5f70fef80 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -113,4 +113,6 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. * Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. -* Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). \ No newline at end of file +* Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). +* Improves Timeline's table performance when row renderers are switched on ({kibana-pull}193316[#193316]). +* Fixes misaligned filter control labels on the Alerts page. \ No newline at end of file From bbfa1789b99db5ce697a16164163fa92552706bd Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Tue, 5 Nov 2024 15:42:48 -0500 Subject: [PATCH 27/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index e5f70fef80..82a561d18b 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -38,7 +38,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). * Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). -* Introduces the ability to add notes to alerts and events, in addition to Timeline. +* Introduces the ability to add notes to alerts and events, in addition to Timeline ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). //PR for notes feature is incoming. * Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) From d5e25e33e1dfdc51e65f9d3735830772a248f1cf Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Wed, 6 Nov 2024 16:25:35 -0500 Subject: [PATCH 28/51] Adding known manual run issues --- docs/release-notes/8.16.asciidoc | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 82a561d18b..2eeaaa9188 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -9,6 +9,28 @@ [[known-issue-8.16.0]] ==== Known issues +// tag::known-issue[] +[discrete] +.Duplicate alerts can be produced from manually running threshold rules +[%collapsible] +==== +*Details* + +On November 12, 2024, it was discovered that manually running threshold rules could produce duplicate alerts if the date range was already covered by a scheduled rule execution. + +==== +// end::known-issue[] + +// tag::known-issue[] +[discrete] +.Tags appear in Elastic AI Assistant's responses +[%collapsible] +==== +*Details* + +On November 12, 2024, it was discovered that manually running a custom query rule with suppression could produce an inflated amount of suppressed alerts. + +==== +// end::known-issue[] + // tag::known-issue-189676[] [discrete] .Tags appear in Elastic AI Assistant's responses @@ -115,4 +137,4 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. * Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). * Improves Timeline's table performance when row renderers are switched on ({kibana-pull}193316[#193316]). -* Fixes misaligned filter control labels on the Alerts page. \ No newline at end of file +* Fixes misaligned filter control labels on the Alerts page ({kibana-pull}192094[#192094]). \ No newline at end of file From df1dac92e5f4f31c0896962522f766a1393ce6c0 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 6 Nov 2024 16:50:59 -0500 Subject: [PATCH 29/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Eric Beahan --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 2eeaaa9188..90a6c9490a 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -87,7 +87,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). * Allows Automatic Import to analyze a larger number of sample events when generating a new integration ({kibana-pull}196233[#196233]). -* Improves Automatic Import's ability to recognize events formatted as CSV ({kibana-pull}196228[#196228]). +* Allows Automatic Import to recognize CSV logs and create integrations for CSV data ({kibana-pull}196228[#196228], {kibana-pull}194386[#194386]). * Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). * Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). * Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]). From 1decd9885ebd8bc6c77a247016cd6c4d4f85c62e Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Wed, 6 Nov 2024 16:55:27 -0500 Subject: [PATCH 30/51] Fix title --- docs/release-notes/8.16.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 2eeaaa9188..16e2daa859 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -22,11 +22,11 @@ On November 12, 2024, it was discovered that manually running threshold rules co // tag::known-issue[] [discrete] -.Tags appear in Elastic AI Assistant's responses +.Manually running custom query rules with suppression could suppress more alerts than expected [%collapsible] ==== *Details* + -On November 12, 2024, it was discovered that manually running a custom query rule with suppression could produce an inflated amount of suppressed alerts. +On November 12, 2024, it was discovered that manually running a custom query rule with suppression could incorrectly inflate the number of suppressed alerts. ==== // end::known-issue[] From 68f56f72be45010a61d7e4c4a11d5eaaeaeecab8 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 6 Nov 2024 17:07:49 -0800 Subject: [PATCH 31/51] Adds knowledge base index known error --- docs/release-notes/8.16.asciidoc | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index b13e2d5143..51e9da9278 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -20,6 +20,21 @@ On November 12, 2024, it was discovered that manually running threshold rules co ==== // end::known-issue[] +// tag::known-issue[] +[discrete] +.Attempting to edit an Elastic AI Assistant Knowledge Base index results in an error +[%collapsible] +==== +*Details* + +Updating a Knowledge Base entry of type "index" results in an error. + +*Workaround* + +Instead of updating an "index" entry, delete it and add it again with the desired changes. + +==== +// end::known-issue[] + + // tag::known-issue[] [discrete] .Manually running custom query rules with suppression could suppress more alerts than expected From 168df5638b66383c07905a95300fb621848c0a4b Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Thu, 7 Nov 2024 10:09:20 -0500 Subject: [PATCH 32/51] Update docs/release-notes/8.16.asciidoc --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 51e9da9278..a739d41eec 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -67,7 +67,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when [[features-8.16.0]] ==== New features -// * Introduces a new API route for listing Entity Store entities: `GET /api/entity_store/entities/list` ({kibana-pull}192806[#192806]). +Introduces the entity store as a technical preview feature, which allows observed, imported, integrated, or uploaded entities to be stored persistently ({kibana-pull}192806[#192806]). * Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]). * Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). From ca9d5ef269a47f7fc0c7cff0a0d1080c9e54ad67 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Thu, 7 Nov 2024 11:38:59 -0500 Subject: [PATCH 33/51] Re-orders new features --- docs/release-notes/8.16.asciidoc | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index a739d41eec..e5095f5ffb 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -67,22 +67,22 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when [[features-8.16.0]] ==== New features -Introduces the entity store as a technical preview feature, which allows observed, imported, integrated, or uploaded entities to be stored persistently ({kibana-pull}192806[#192806]). * Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]). -* Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). +* Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). +* Adds ability to manually run for a specified time period, either for testing purposes or to generate alerts for past events. +* Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). +* Introduces the ability to add notes to alerts and events, in addition to Timeline ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). +* Enables detection rules to automatically execute system actions, such as opening a case ({kibana-pull}183937[#183937]). +* Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). +* Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). +* Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]). +* Introduces the entity store as a technical preview feature, which allows observed, imported, integrated, or uploaded entities to be stored persistently ({kibana-pull}192806[#192806]). * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). * Provides a way to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). * Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). -* Introduces the ability to add notes to alerts and events, in addition to Timeline ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). -//PR for notes feature is incoming. * Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). -* Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]) -* Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). -* Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). -* Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). -* Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. * To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. @@ -111,7 +111,7 @@ Introduces the entity store as a technical preview feature, which allows observe ** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} ** Attack Discovery now combines related discoveries that would previously have appeared separately ** Attack Discovery now detects and displays an error instead of hallucinated output -* Adds an **Install and enable** button to the **Add Elastic Rules** page, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). +* Adds an **Install and enable** button to the **Add Elastic Rules** page, which allows for rules to be immediately enabled after they're installed ({kibana-pull}191529[#191529]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). * Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). From 02fc62e042734b28ca8109f505f78875ba4244a6 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Thu, 7 Nov 2024 16:11:14 -0500 Subject: [PATCH 34/51] Edits and summary for 191874 --- docs/release-notes/8.16.asciidoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index e5095f5ffb..b8df4f093b 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -73,7 +73,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). * Introduces the ability to add notes to alerts and events, in addition to Timeline ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). * Enables detection rules to automatically execute system actions, such as opening a case ({kibana-pull}183937[#183937]). -* Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). +* Adds role-based acccess control (RBAC) for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]). * Introduces the entity store as a technical preview feature, which allows observed, imported, integrated, or uploaded entities to be stored persistently ({kibana-pull}192806[#192806]). @@ -97,7 +97,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when [discrete] [[enhancements-8.16.0]] ==== Enhancements -* Enables you to open the Rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). +* Enables you to open the rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). * Allows you to resize the alert and event details flyouts and choose how it's displayed (over the Alerts table or next to it) ({kibana-pull}192906[#192906], {kibana-pull}182615[#182615]). * Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). * Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). @@ -114,12 +114,13 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Adds an **Install and enable** button to the **Add Elastic Rules** page, which allows for rules to be immediately enabled after they're installed ({kibana-pull}191529[#191529]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). -* Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). +* Adds the **Alert Suppression** and **Investigative guide** fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). * Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint details flyout ({kibana-pull}184125[#184125]). * Allows you to recalculate entity risk scores immediately after you upload asset criticality data ({kibana-pull}187577[#187577]). * Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Improves network previews in the alert details flyout ({kibana-pull}190560[#190560]). +* Adds support in all detection rule types for {elastic-defend}'s automated response actions ({kibana-pull}193390[#193390], {kibana-pull}191874[#191874]). * Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. * Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`. * Adds a new {elastic-defend} API event for https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol[`DeviceIoControl`] calls to support the detection of driver abuse. This feature is only supported on Windows 11 Desktop versions. @@ -140,8 +141,8 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Prevents an empty warning message from appearing for rule executions ({kibana-pull}186096[#186096]). * Fixes an error that could occur during rule execution when the source index had a non-ECS-compliant text field ({kibana-pull}187673[#187673]). * Removes unnecessary empty space below the title of the Open Timeline modal ({kibana-pull}188837[#188837]). -* Improves the Alerts table's performance ({kibana-pull}192827[#192827]). -* Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]). +* Improves the performance of the Alerts table ({kibana-pull}192827[#192827]). +* Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture findings ({kibana-pull}194069[#194069]). * Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). * Fixes an {elastic-defend} bug where network event deduplication logic could incorrectly drop Linux network events. * Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. From d9c762e8e710b2e3575914e914d348439d2deb16 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 8 Nov 2024 12:28:30 -0500 Subject: [PATCH 35/51] Grammar and re-orders enh and bf --- docs/release-notes/8.16.asciidoc | 46 ++++++++++++++++---------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index b8df4f093b..d29bc64f43 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -61,7 +61,7 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when [[breaking-changes-8.16.0]] ==== Breaking changes -* During shutdown, {kib} now waits for all the ongoing requests to complete according to the `server.shutdownTimeout` setting. During that period, the incoming socket is closed and any new incoming requests are rejected. Before this update, new incoming requests received a response with the status code 503 and body { "message": "{kib} is shutting down and not accepting new incoming requests" }. +* During shutdown, {kib} now waits for all the ongoing requests to complete according to the `server.shutdownTimeout` setting. During that period, the incoming socket is closed and any new incoming requests are rejected. Before this update, new incoming requests received a response with the status code 503 and body `{ "message": "{kib} is shutting down and not accepting new incoming requests" }`. [discrete] [[features-8.16.0]] @@ -69,11 +69,11 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]). * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). -* Adds ability to manually run for a specified time period, either for testing purposes or to generate alerts for past events. +* Adds ability to manually run rules for a specified time period, either for testing purposes or to generate alerts for past events. * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). -* Introduces the ability to add notes to alerts and events, in addition to Timeline ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). +* Adds the ability to attach notes to alerts and events and introduces the Notes page, which allows you to manage all existing notes ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). * Enables detection rules to automatically execute system actions, such as opening a case ({kibana-pull}183937[#183937]). -* Adds role-based acccess control (RBAC) for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). +* Adds role-based access control (RBAC) for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]). * Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]). * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]). * Introduces the entity store as a technical preview feature, which allows observed, imported, integrated, or uploaded entities to be stored persistently ({kibana-pull}192806[#192806]). @@ -82,43 +82,43 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). * Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). -* Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). +* Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. * To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. * You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: -** This can greatly increase {elastic-defend}'s CPU and I/O utilization, and impact system responsiveness. -** This can significantly delay event enrichment, and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. -** This can cause event processing queues to overflow, and lead to dropped events. -** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing]. -* Improves {elastic-defend} enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the kafka output. +** This can greatly increase {elastic-defend}'s CPU and I/O utilization and impact system responsiveness. +** This can significantly delay event enrichment and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. +** This can cause event processing queues to overflow and lead to dropped events. +** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing]. +* Improves {elastic-defend} enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the Kafka output. * Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. [discrete] [[enhancements-8.16.0]] ==== Enhancements -* Enables you to open the rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). -* Allows you to resize the alert and event details flyouts and choose how it's displayed (over the Alerts table or next to it) ({kibana-pull}192906[#192906], {kibana-pull}182615[#182615]). -* Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). -* Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). -* Allows Automatic Import to analyze a larger number of sample events when generating a new integration ({kibana-pull}196233[#196233]). -* Allows Automatic Import to recognize CSV logs and create integrations for CSV data ({kibana-pull}196228[#196228], {kibana-pull}194386[#194386]). * Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). * Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). * Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]). -* Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). +* Adds an **Other** option to the OpenAI connector's **Select an OpenAI provider** dropdown menu. Select this option when <> ({kibana-pull}194831[#194831]). +* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). +* Adds an **Install and enable** button to the **Add Elastic Rules** page, which allows for rules to be immediately enabled after they're installed ({kibana-pull}191529[#191529]). +* Adds the **Alert Suppression** and **Investigative guide** fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). +* Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Improves Attack Discovery in the following ways ({kibana-pull}195669[#195669]): -** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es} +** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page and is stored locally instead of in {es} ** Attack Discovery now combines related discoveries that would previously have appeared separately ** Attack Discovery now detects and displays an error instead of hallucinated output -* Adds an **Install and enable** button to the **Add Elastic Rules** page, which allows for rules to be immediately enabled after they're installed ({kibana-pull}191529[#191529]). * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). -* Adds the **Alert Suppression** and **Investigative guide** fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). * Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint details flyout ({kibana-pull}184125[#184125]). * Allows you to recalculate entity risk scores immediately after you upload asset criticality data ({kibana-pull}187577[#187577]). -* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). -* Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). +* Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]). +* Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). +* Allows Automatic Import to analyze a larger number of sample events when generating a new integration ({kibana-pull}196233[#196233]). +* Allows Automatic Import to recognize CSV logs and create integrations for CSV data ({kibana-pull}196228[#196228], {kibana-pull}194386[#194386]). +* Allows you to open the rule details flyout from the Alerts table ({kibana-pull}191764[#191764]). +* Allows you to resize the alert and event details flyouts and choose how it's displayed in relation to the Alerts table (over or next to it) ({kibana-pull}192906[#192906], {kibana-pull}182615[#182615]). * Improves network previews in the alert details flyout ({kibana-pull}190560[#190560]). * Adds support in all detection rule types for {elastic-defend}'s automated response actions ({kibana-pull}193390[#193390], {kibana-pull}191874[#191874]). * Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis. @@ -140,10 +140,10 @@ On August 1, 2024, it was discovered that Elastic AI Assistant's responses when * Prevents an empty warning message from appearing for rule executions ({kibana-pull}186096[#186096]). * Fixes an error that could occur during rule execution when the source index had a non-ECS-compliant text field ({kibana-pull}187673[#187673]). +* Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). * Removes unnecessary empty space below the title of the Open Timeline modal ({kibana-pull}188837[#188837]). * Improves the performance of the Alerts table ({kibana-pull}192827[#192827]). * Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture findings ({kibana-pull}194069[#194069]). -* Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]). * Fixes an {elastic-defend} bug where network event deduplication logic could incorrectly drop Linux network events. * Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. * Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields. From cac4ce41231addc1e5441d2352581e47befce4f8 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 8 Nov 2024 12:30:02 -0500 Subject: [PATCH 36/51] Re-orders known issues --- docs/release-notes/8.16.asciidoc | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index d29bc64f43..8330e5bfe4 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -11,51 +11,50 @@ // tag::known-issue[] [discrete] -.Duplicate alerts can be produced from manually running threshold rules +.Attempting to edit an Elastic AI Assistant Knowledge Base index results in an error [%collapsible] ==== *Details* + -On November 12, 2024, it was discovered that manually running threshold rules could produce duplicate alerts if the date range was already covered by a scheduled rule execution. +Updating a Knowledge Base entry of type "index" results in an error. + +*Workaround* + +Instead of updating an "index" entry, delete it and add it again with the desired changes. ==== // end::known-issue[] -// tag::known-issue[] +// tag::known-issue-189676[] [discrete] -.Attempting to edit an Elastic AI Assistant Knowledge Base index results in an error +.Tags appear in Elastic AI Assistant's responses [%collapsible] ==== *Details* + -Updating a Knowledge Base entry of type "index" results in an error. - -*Workaround* + -Instead of updating an "index" entry, delete it and add it again with the desired changes. +On August 1, 2024, it was discovered that Elastic AI Assistant's responses when using Bedrock Sonnet 3.5 may include `` tags, for example `` ({kibana-issue}189676[#189676]). ==== -// end::known-issue[] - +// end::known-issue-189676[] // tag::known-issue[] [discrete] -.Manually running custom query rules with suppression could suppress more alerts than expected +.Duplicate alerts can be produced from manually running threshold rules [%collapsible] ==== *Details* + -On November 12, 2024, it was discovered that manually running a custom query rule with suppression could incorrectly inflate the number of suppressed alerts. +On November 12, 2024, it was discovered that manually running threshold rules could produce duplicate alerts if the date range was already covered by a scheduled rule execution. ==== // end::known-issue[] -// tag::known-issue-189676[] +// tag::known-issue[] [discrete] -.Tags appear in Elastic AI Assistant's responses +.Manually running custom query rules with suppression could suppress more alerts than expected [%collapsible] ==== *Details* + -On August 1, 2024, it was discovered that Elastic AI Assistant's responses when using Bedrock Sonnet 3.5 may include `` tags, for example `` ({kibana-issue}189676[#189676]). +On November 12, 2024, it was discovered that manually running a custom query rule with suppression could incorrectly inflate the number of suppressed alerts. ==== -// end::known-issue-189676[] +// end::known-issue[] [discrete] [[breaking-changes-8.16.0]] From 4ea3583e457aaaa9fbe9cba4b9852de3b1ec8529 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 8 Nov 2024 15:13:14 -0500 Subject: [PATCH 37/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Joe Peeples --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 8330e5bfe4..a1c287e4f1 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -90,7 +90,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul ** This can significantly delay event enrichment and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. ** This can cause event processing queues to overflow and lead to dropped events. ** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing]. -* Improves {elastic-defend} enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the Kafka output. +* Improves {elastic-defend} by enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the Kafka output. * Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. [discrete] From d1339a9280048816966a85c0ca99b773b4d5d3fe Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 8 Nov 2024 15:13:21 -0500 Subject: [PATCH 38/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Joe Peeples --- docs/release-notes/8.16.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index a1c287e4f1..500550b9cd 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -105,9 +105,9 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Adds the **Alert Suppression** and **Investigative guide** fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). * Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). * Improves Attack Discovery in the following ways ({kibana-pull}195669[#195669]): -** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page and is stored locally instead of in {es} -** Attack Discovery now combines related discoveries that would previously have appeared separately -** Attack Discovery now detects and displays an error instead of hallucinated output +** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page and is stored locally instead of in {es}. +** Attack Discovery now combines related discoveries that would previously have appeared separately. +** Attack Discovery now detects and displays an error instead of hallucinated output. * Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). * Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). * Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint details flyout ({kibana-pull}184125[#184125]). From 8f8158a0824138f4f1e26074fbfb97c40ea9dcc1 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 8 Nov 2024 15:13:31 -0500 Subject: [PATCH 39/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Joe Peeples --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 500550b9cd..978bacc73c 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -147,7 +147,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. * Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields. * Fixes a bug where {elastic-defend} could fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. -* Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. +* Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} to use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. * Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. * Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). From 3d43f1d8fa69d9ba86383ee5749b36b0c10a6204 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 8 Nov 2024 15:13:38 -0500 Subject: [PATCH 40/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Joe Peeples --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 978bacc73c..f81dc11d2e 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -150,6 +150,6 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} to use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. * Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. -* Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). +* Fixes scenarios where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). * Improves Timeline's table performance when row renderers are switched on ({kibana-pull}193316[#193316]). * Fixes misaligned filter control labels on the Alerts page ({kibana-pull}192094[#192094]). \ No newline at end of file From 779327cf3dab890ed70dccfebc26b8d80e0758fc Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Mon, 11 Nov 2024 10:00:19 -0800 Subject: [PATCH 41/51] adds cloud sec integrations --- docs/release-notes/8.16.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index f81dc11d2e..fb3535b78e 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -68,6 +68,8 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]). * Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). +* Enables data collected by the Wiz and AWS Security Hub integrations to appear on the Findings page and in entity details flyouts (https://github.com/elastic/integrations/pull/10790[#10790], https://github.com/elastic/integrations/pull/11158[#11158]). +* Enables alerts collected by the Falco integration to appear on the Alerts page (https://github.com/elastic/integrations/pull/9619[#9619], https://github.com/elastic/integrations/pull/11051[#11051]). * Adds ability to manually run rules for a specified time period, either for testing purposes or to generate alerts for past events. * Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). * Adds the ability to attach notes to alerts and events and introduces the Notes page, which allows you to manage all existing notes ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]). From 62adef9b862b45426c7d68caab26d15780adbfcf Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:49:17 -0500 Subject: [PATCH 42/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index fb3535b78e..552b00a46c 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -79,7 +79,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <> workflows by default ({kibana-pull}196270[#196270]). * Introduces the entity store as a technical preview feature, which allows observed, imported, integrated, or uploaded entities to be stored persistently ({kibana-pull}192806[#192806]). * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). -* Provides a way to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). +* Allows you to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). * Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). From 5df7dc7522dc4ff690bd318c48833b558cdd91aa Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:49:28 -0500 Subject: [PATCH 43/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 552b00a46c..0e9728894f 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -81,7 +81,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]). * Allows you to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]). * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]). -* Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). +* Introduces a new advanced setting, `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). * Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. From 9e18601f81b6e8f5cf412585d57d51ca07645010 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:49:37 -0500 Subject: [PATCH 44/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 0e9728894f..29e4839cc8 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -85,7 +85,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). * Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. -* To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. +* Allows you to reduce CPU usage, I/O, and event sizes by turning on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. * To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. * You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: ** This can greatly increase {elastic-defend}'s CPU and I/O utilization and impact system responsiveness. From e3954a53f0a4308277a545d5ccc3ae442d1e7098 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:49:45 -0500 Subject: [PATCH 45/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 29e4839cc8..6bcd898b1f 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -86,7 +86,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * Allows you to reduce CPU usage, I/O, and event sizes by turning on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. -* To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. +* Allows you to reduce CPU usage, I/O, and event sizes by turning off MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. * You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: ** This can greatly increase {elastic-defend}'s CPU and I/O utilization and impact system responsiveness. ** This can significantly delay event enrichment and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. From 95676b38dfe045aae9dbf05f3915f9d3726cdcd8 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:49:54 -0500 Subject: [PATCH 46/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 6bcd898b1f..c9408bf180 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -87,7 +87,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * Allows you to reduce CPU usage, I/O, and event sizes by turning on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. * Allows you to reduce CPU usage, I/O, and event sizes by turning off MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. -* You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: +* Allows you to configure your {elastic-defend} integration policy to collect SHA-256 file hashes in file events. Before doing so, consider the following caveats: ** This can greatly increase {elastic-defend}'s CPU and I/O utilization and impact system responsiveness. ** This can significantly delay event enrichment and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. ** This can cause event processing queues to overflow and lead to dropped events. From eae9b8fb3f5f3966ffee4ee7396dee734d89a956 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:50:01 -0500 Subject: [PATCH 47/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index c9408bf180..5a2a2b2449 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -130,7 +130,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Ensures that {elastic-defend} tells {fleet} that it's `orphaned` if the connection between {elastic-defend} and {agent} stops for an extended period of time. {fleet} uses this information to provide you with additional troubleshooting context. * Adds SOCKS5 proxy support to {elastic-defend}'s {ls} output. * Ensures that on Windows, {elastic-defend} uses https://www.elastic.co/security-labs/finding-truth-in-the-shadows[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables the detection of certain defense evasions. You can turn this feature off in {elastic-defend} <> ({kibana-pull}190553[#190553]). -* Restore {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0. +* Restores {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0. * Improves {elastic-defend}'s caching to reduce memory usage on Windows. * Enhances {elastic-defend} by reducing the size of process events, which reduces excessive process ancestry entries and shortens the entity ID. * Improves the reliability and system resource usage of {elastic-defend}'s Windows network driver. From c1e78ba33c16d5be51ba7f9ed6d64ec8643eb6ee Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:50:13 -0500 Subject: [PATCH 48/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 5a2a2b2449..cba8e78f02 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -84,7 +84,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Introduces a new advanced setting, `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]). * Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). * Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]). -* The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. +* Turns off the host field size reduction setting on {elastic-defend}'s integration policy by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <>. * Allows you to reduce CPU usage, I/O, and event sizes by turning on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <>. * Allows you to reduce CPU usage, I/O, and event sizes by turning off MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. * Allows you to configure your {elastic-defend} integration policy to collect SHA-256 file hashes in file events. Before doing so, consider the following caveats: From 2e2d9a71cb0d9bc926d7994c8dcdb4079ff8ee6a Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:50:27 -0500 Subject: [PATCH 49/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index cba8e78f02..d6c78163ef 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -91,7 +91,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul ** This can greatly increase {elastic-defend}'s CPU and I/O utilization and impact system responsiveness. ** This can significantly delay event enrichment and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior. ** This can cause event processing queues to overflow and lead to dropped events. -** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing]. +** Many file events won't contain hashes. Hash collection is the best effort and is not guaranteed to be present in every event. Hashes are collected asynchronously and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing]. * Improves {elastic-defend} by enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the Kafka output. * Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity. From 28afc0d0306e7bdc4884d3ccd04b12f53cc2fecc Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:50:36 -0500 Subject: [PATCH 50/51] Update docs/release-notes/8.16.asciidoc Co-authored-by: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index d6c78163ef..4041908314 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -149,7 +149,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul * Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI. * Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields. * Fixes a bug where {elastic-defend} could fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. -* Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} to use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. +* Fixes a bug that prevented host name uniformity with {beats} products. If you request {elastic-defend} to use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default. * Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts. * Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy. * Fixes scenarios where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]). From ff326f6267d7cdc99e78ac4f0963cb73f9719641 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Mon, 11 Nov 2024 15:31:53 -0500 Subject: [PATCH 51/51] Updates summary for 191557 --- docs/release-notes/8.16.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc index 4041908314..6e290fc436 100644 --- a/docs/release-notes/8.16.asciidoc +++ b/docs/release-notes/8.16.asciidoc @@ -67,7 +67,7 @@ On November 12, 2024, it was discovered that manually running a custom query rul ==== New features * Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]). -* Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]). +* Enables agentless deployment for Elastic's Cloud Security Posture Management integration and the new Cloud Asset Inventory integration ({kibana-pull}191557[#191557]). * Enables data collected by the Wiz and AWS Security Hub integrations to appear on the Findings page and in entity details flyouts (https://github.com/elastic/integrations/pull/10790[#10790], https://github.com/elastic/integrations/pull/11158[#11158]). * Enables alerts collected by the Falco integration to appear on the Alerts page (https://github.com/elastic/integrations/pull/9619[#9619], https://github.com/elastic/integrations/pull/11051[#11051]). * Adds ability to manually run rules for a specified time period, either for testing purposes or to generate alerts for past events.