[Security Solution] Adds customized_fields and has_base_version fields to rule_source object schema

[dplumlee]

Resolves: https://github.com/elastic/security-team/issues/12507 (internal)

Summary

Adds two new fields to the existing rule_source object in our rule schema as described in #230856. Also updates and adds test coverage for the new field logic.

The new fields are:

• customized_fields: an array of objects containing rule field names that have been modified from the base version of the prebuilt rule.
  • Defaults to empty array if prebuilt rule is not customized or if base version did not exist during diff calculation.
• has_base_version: a boolean field that specifies if the base version of a prebuilt rule was able to be fetched and used during the customization calculation.

This PR also adds related telemetry fields as described in #230856. This includes a customizations object field which contains a slimmed down version of customized_fields and has a num_functional_fields number field that is created in the telemetry task pipeline by comparing the customized fields array to a constant list of field names that we are defining as "functional". This source of truth list can be found in the x-pack/solutions/security/plugins/security_solution/common/detection_engine/constants.ts file

Examples

{
  "rule_source": {
    "type": "external",
    "is_customized": true,
    /* New fields */
    "customized_fields": [
      {
        "field_name": "tags",
      },
      {
        "field_name": "query",
      }
    ],
    "has_base_version": true
  }
}

"customizations": {
  "customized_fields": ["tags", "query"],
  "num_functional_fields": 2,
}

How to test telemetry

Link to internal staging with example data: (internal staging)

1. Set the prebuilt rule task type to something shorter than 1hr in this file: x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/tasks/prebuilt_rule_alerts.ts
2. Add the following to kibana.dev.yml:

telemetry.enabled: true
telemetry.optIn: true

// (Optional for checking to see if its working)
logging:
  root:
    appenders: [default]
    level: warn
  loggers:
    - name: plugins.securitySolution
      level: debug
    - name: plugins.ruleRegistry
    - name: plugins.taskManager

3. Start up both Elasticsearch and kibana (Has to be done after updating task interval as task objects are stored in ES)
4. Install prebuilt rules
5. Modify prebuilt rules with different field customizations and enable those rules
6. Generate alerts that match these rules (resolver script generator, dev tools, query modification, etc.)
7. View the alerts getting sent to the internal staging telemetry cluster (https://analytics-staging.sde.elastic.dev) in the detections_alert_telemetry_elastic* index
8. Use the new customizations field to filter out/in customized rule alerts

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
  • Rule customization tests

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[jkelas]

Hi @dplumlee
I reviewed your changes and the code looks very good. Well done.
I only left two minor comments.

I did the manual testing and all works fine with the modified functionality. I also observed the changes made in the kibana_alerting_cases index for modified rules and everything worked as expected.

[jkelas]

Thank you @dplumlee for adressing my comments.
I am approving.

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9317

[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/configs/ess.config.ts: 100/100 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/configs/serverless.config.ts: 100/100 tests passed.

see run history

[natasha-moore-elastic]

Left some optional wording suggestions, otherwise LGTM!

[banderror]

@dplumlee This fixed implementation looks correct to me, thanks for the fixes and the additional test coverage 👍

I left just a couple nits for your consideration.

I'm trying to test this locally, just a happy path. I can confirm that a customized rule gets a rule_source with all the fields correctly initialized. I can also see that telemetry is being sent. But I'm not seeing it in the analytics-staging cluster. How much time does it take to process and index the incoming telemetry events? Is there any sampling that requires sending a lot of it so that at least some gets indexed?

Approving the PR in advance because the thoroughly reviewed implementation looks correct and the test coverage seems robust enough. Thanks for the hard work @dplumlee!

[dplumlee]

@banderror It wasn't immediately indexed for me when I was testing this but always fairly quick (~5-10 min) when I updated the task schedule and restarted ES locally. Perhaps I was lucky in my timing?

Is there any sampling that requires sending a lot of it so that at least some gets indexed?

I was sending about a 100 alerts per rule, mostly from the resolver generator script but I don't believe the amount of data should matter, there's nothing in the tasks that limit this behavior

[banderror]

@dplumlee It still hasn't indexed anything from my machine. I hope I did something wrong, but we should double-check before merging. Let's test it live at our tech sync today.

[banderror]

@elasticmachine merge upstream

[jkelas]

@dplumlee I tested that after your fix of the pipeline the events reach staging correctly, I was able to see events generated by my customized rule, e.g.

 "customizations": {
      "customized_fields": [
        "name",
        "tags",
        "related_integrations",
        "index",
        "filters"
      ],
      "num_functional_fields": 1
    },

[kibanamachine]

Starting backport for target branches: 9.1, 9.2

https://github.com/elastic/kibana/actions/runs/18232333038

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5079038

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.1MB	11.1MB	+110.0B

History

• 💔 Build #347161 failed 7aa1ad7
• 💛 Build #346668 was flaky 8eab266
• 💚 Build #346471 succeeded 1caa297
• 💔 Build #346338 failed cb163a4
• 💔 Build #346049 failed 664aca0
• 💔 Build #346004 failed 14d09a9

cc @dplumlee

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation