Skip to content

[9.1] [Security Solution] Two-way prebuilt rule diff (#233045)#234707

Merged
kibanamachine merged 1 commit intoelastic:9.1from
kibanamachine:backport/9.1/pr-233045
Sep 11, 2025
Merged

[9.1] [Security Solution] Two-way prebuilt rule diff (#233045)#234707
kibanamachine merged 1 commit intoelastic:9.1from
kibanamachine:backport/9.1/pr-233045

Conversation

@kibanamachine
Copy link
Contributor

Backport

This will backport the following commits from main to 9.1:

Questions ?

Please refer to the Backport tool documentation

**Partially addresses:**
elastic/security-team#12507 (internal)

## Summary

Adds a 2-way diff comparison algorithm to the codebase and shifts the
existing diff code to a "three-way" naming scheme.

In order for the telemetry work described in
elastic#230856 to occur, we need to
refactor our diff calculation method to return rule schema fields
instead of the `DiffableRule` schema that we currently return. This
required a lot of exploratory work to determine the best method in which
to accomplish this refactor, as the core function,
`calculateRuleDiffFields`, is used as a core piece of logic in all our
prebuilt rule customization related endpoints, and most of our UI
components are reliant on this `DiffableRule` based return structure.

I settled on an approach that adds a separate 2-way diff function that
never converts the rule into the diffable rule structure, allowing us to
directly compare the rule objects and return exact rule schema fields.
This also involved refactoring the normalization process we did in
`convertToDiffable` and extracting out the comparison functions we used
in the diff algorithms, so that both diff algorithms would be returning
identical "is_customized" calculations given a set of `RuleResponse`
objects.

This PR adds a comprehensive set of tests that verify the
synchronization of these two diff algorithms for ease of maintainability
and to greatly lessen the possibility of divergence.

This PR also swaps out the `is_customized` rule source calculation with
the new 2-way diff calculation, all tests are running (and passing) with
this new diff calculation.

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] [FTR tests
run](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9301)
- [x] [Cypress tests
run](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9302)

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit b08abf7)
@kibanamachine kibanamachine added the backport This PR is a backport of another PR label Sep 11, 2025
@kibanamachine kibanamachine enabled auto-merge (squash) September 11, 2025 06:08
@kibanamachine kibanamachine merged commit 1ea4839 into elastic:9.1 Sep 11, 2025
16 checks passed
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 7817 7821 +4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 9.9MB 9.9MB +3.0KB

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id before after diff
_data_stream_timestamp 1 - -1
_doc_count 1 - -1
_ignored_source 1 - -1
_index_mode 1 - -1
_inference_fields 1 - -1
_tier 1 - -1
apm-custom-dashboards 5 - -5
apm-server-schema 2 - -2
apm-service-group 5 - -5
application_usage_daily 2 - -2
config 2 - -2
config-global 2 - -2
coreMigrationVersion 1 - -1
created_at 1 - -1
created_by 1 - -1
entity-definition 9 - -9
entity-discovery-api-key 2 - -2
event_loop_delays_daily 2 - -2
favorites 4 - -4
file 11 - -11
file-upload-usage-collection-telemetry 3 - -3
fileShare 5 - -5
guided-onboarding-guide-state 3 - -3
infra-custom-dashboards 4 - -4
infrastructure-monitoring-log-view 2 - -2
intercept_trigger_record 5 - -5
legacy-url-alias 7 - -7
managed 1 - -1
ml-job 6 - -6
ml-module 13 - -13
ml-trained-model 7 - -7
monitoring-telemetry 2 - -2
namespace 1 - -1
namespaces 1 - -1
observability-onboarding-state 2 - -2
originId 1 - -1
product-doc-install-status 7 - -7
references 4 - -4
sample-data-telemetry 3 - -3
security-ai-prompt 8 - -8
slo 11 - -11
space 5 - -5
synthetics-monitor 34 - -34
synthetics-monitor-multi-space 34 - -34
tag 4 - -4
type 1 - -1
typeMigrationVersion 1 - -1
ui-metric 2 - -2
updated_at 1 - -1
updated_by 1 - -1
upgrade-assistant-ml-upgrade-operation 3 - -3
upgrade-assistant-reindex-operation 3 - -3
uptime-synthetics-api-key 2 - -2
url 5 - -5
usage-counters 2 - -2
total -249

cc @dplumlee

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants