[Security Solution] Allow users to edit related_integrations field for custom rules

[approksiu]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

We need to expose the related_integrations field in the rule edit page and allow editing it.
We should provide a list of valid security integrations for the users to choose from.
It is an optional field.

Background

We want to allow users to easily specify the prerequisites for their custom rules.

User story

• As a user, I want to be able to specify related integrations fields for my custom rules.

Acceptance criteria

• Rule edit page shows the related_integrations field in the Define Rule section.
• User can select the related integrations from a list of available security integrations.
• On the rule edit page, there is a tooltip explaining what this field does, (and a link to documentation?).
• User can save the rule without this field specified.
• When user is duplicating a prebuilt rule, the related integration field should be kept.

Designs

Figma file

Release progress

• Initial implementation is done. (PR)
• Feature is covered with automated tests.
• Feature is fully implemented and considered by the development team as ready to be released.
• Acceptance testing is done and the feature is approved by @approksiu and @ARWNightingale.
• Exploratory testing is done and the feature is approved by @vgomez-el.
• Documentation is written for ESS and Serverless by @joepeeples. Three docs PRs are approved and ready to be merged. (ticket)
• Feature is released in Serverless.

Planned release date in Serverless: Week of May 6, 2024.
Planned release date in ESS: June 18 2024 (v8.15.0).

[jpdjere]

The related integrations field has the schema:

    RelatedIntegrationArray:
      type: array
      items:
        $ref: '#/components/schemas/RelatedIntegration'

    RelatedIntegration:
      type: object
      properties:
        package:
          $ref: '#/components/schemas/NonEmptyString'
        version:
          $ref: '#/components/schemas/NonEmptyString'
        integration:
          $ref: '#/components/schemas/NonEmptyString'
      required:
        - package
        - version

for example, for existing rules:

[
  {
    "name": "Linux Restricted Shell Breakout via Linux Binary(s)",
    "ri": [{ "package": "endpoint", "version": "^8.2.0" }]
  },
  {
    "name": "Potential Credential Access via DCSync",
    "ri": [
      { "package": "system", "version": "^1.6.4" },
      { "package": "windows", "version": "^1.5.0" }
    ]
  },
  {
    "name": "Microsoft Build Engine Using an Alternate Name",
    "ri": [
      { "package": "endpoint", "version": "^8.2.0" },
      { "package": "windows", "version": "^1.5.0" }
    ]
  },

  {
    "name": "Unusual AWS Command for a User",
    "ri": [{ "package": "aws", "version": "^2.0.0" }]
  },
  {
    "name": "First Time Seen AWS Secret Value Accessed in Secrets Manager",
    "ri": [
      { "package": "aws", "version": "^2.0.0", "integration": "cloudtrail" }
    ]
  },
  {
    "name": "AWS S3 Bucket Configuration Deletion",
    "ri": [
      { "package": "aws", "version": "^2.0.0", "integration": "cloudtrail" }
    ]
  }
]

Notice that the integration field is optional, and indeed some have it and some not. The integration field supercedes the package field what we show in the UI. For example, for the related integration:

 { "package": "aws", "version": "^2.0.0" }

We link to the page: /app/integrations/detail/aws-2.0.0/overview

while for the integration:

 { "package": "aws", "version": "^2.0.0", "integration": "cloudtrail" }

we link to: /app/integrations/detail/aws-2.0.0/overview?integration=cloudtrail

We can fetch all packages via the Fleet endpoint GET /kbn/api/fleet/epm/packages, and the response looks like:

{
    "items": [
        {
            "name": "aws",
            "title": "AWS",
            "version": "2.11.1",
            "release": "ga",
            "description": "Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.",
            "type": "integration",
            "download": "/epr/aws/aws-2.11.1.zip",
            "path": "/package/aws/2.11.1",
            "icons": [
                {
                    "src": "/img/logo_aws.svg",
                    "path": "/package/aws/2.11.1/img/logo_aws.svg",
                    "title": "logo aws",
                    "size": "32x32",
                    "type": "image/svg+xml"
                }
            ],
            "policy_templates": [
                // more ingration policies
                {
                    "name": "cloudtrail",
                    "title": "AWS CloudTrail",
                    "description": "Collect AWS CloudTrail logs with Elastic Agent",
                    "icons": [
                        {
                            "src": "/img/logo_cloudtrail.svg",
                            "path": "/package/aws/2.11.1/img/logo_cloudtrail.svg",
                            "title": "AWS Cloudtrail logo",
                            "size": "32x32",
                            "type": "image/svg+xml"
                        }
                    ],
                    "categories": [
                        "security"
                    ]
                },
                {
                    "name": "cloudwatch",
                    "title": "AWS CloudWatch",
                    "description": "Use this integration to collect logs and metrics from Amazon CloudWatch with Elastic Agent, where no out of the box integration is available.",
                    "icons": [
                        {
                            "src": "/img/logo_cloudwatch.svg",
                            "path": "/package/aws/2.11.1/img/logo_cloudwatch.svg",
                            "title": "AWS CloudWatch logo",
                            "size": "32x32",
                            "type": "image/svg+xml"
                        }
                    ],
                    "categories": [
                        "observability",
                        "monitoring"
                    ]
                },
                // more ingration policies
            ],
            "conditions": {
                "kibana": {
                    "version": "^8.10.2"
                },
                "elastic": {
                    "subscription": "basic"
                }
            },
            "owner": {
                "type": "elastic",
                "github": "elastic/obs-ds-hosted-services"
            },
            "categories": [
                "aws"
            ],
            "signature_path": "/epr/aws/aws-2.11.1.zip.sig",
            "id": "aws",
            "status": "not_installed"
        },
        // more integration packages
        {
            "name": "windows",
            "title": "Windows",
            "version": "1.43.0",
            "release": "ga",
            "description": "Collect logs and metrics from Windows OS and services with Elastic Agent.",
            "type": "integration",
            "download": "/epr/windows/windows-1.43.0.zip",
            "path": "/package/windows/1.43.0",
            "icons": [
                {
                    "src": "/img/logo_windows.svg",
                    "path": "/package/windows/1.43.0/img/logo_windows.svg",
                    "title": "logo windows",
                    "size": "32x32",
                    "type": "image/svg+xml"
                }
            ],
            "policy_templates": [
                {
                    "name": "windows",
                    "title": "Windows logs and metrics",
                    "description": "Collect logs and metrics from Windows instances"
                }
            ],
            "conditions": {
                "kibana": {
                    "version": "^8.8.0"
                }
            },
            "owner": {
                "github": "elastic/elastic-agent-data-plane"
            },
            "categories": [
                "os_system",
                "security"
            ],
            "signature_path": "/epr/windows/windows-1.43.0.zip.sig",
            "id": "windows",
            "status": "not_installed"
        },
        
    ]
}

So:

• the top level objects correspond to packages, which have info that we might need such as version and description.
• each of those packages has a policy_templates array whose elements correspond to integrations in our schema. For example, AWS Cloudtrail.

Open questions:

• How should the editing component for this field look like?
• How do we handle the version? Do we just automatically select the previous minor version, major version? Do we allow the user to type in what they want? (This is important for the Related Integration component, where we show if the installation is installed or not)

[approksiu]

For installed integrations - select current version, for non-installed - latest version.
Do we show installed and non-installed integrations, and do we show only security category or others too?

[approksiu]

Consider showing packages and integrations in the "flat" drop-down.

[approksiu]

Do we show installed and non-installed integrations, and do we show only security category or others too?

We should show all security integrations. Would be nice to indicate which ones are installed/not installed - maybe as a follow up? cc @ARWNightingale
It should be possible to search by name

[jpdjere]

Great @approksiu .

So we just met with @ARWNightingale and discussed this.

I think it makes sense to give the user a dropdown of all possible related integrations, pulling the list from Fleet. At the same time, we should show the user which of those integrations are installed, with some kind of label on each integration's row, if it is actually installed. Maybe Alex you can think an example of how this should look like.

Also, integrations that are installed should be pushed up to the top of the list, because we can presume that the user will be selecting one of those with much higher probability.

Makes sense to you @approksiu ?

[approksiu]

@jpdjere totally, thanks!

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[approksiu]

@maximpn my preference is Design 2 option from the figma file. Let's pre-fill version value if possible for the user to accept or modify. Thanks!

[maximpn]

@approksiu thank you for an update 👍

I attach the current Design 2 Figma snapshot for quick reference:

[banderror]

@maximpn Since #178295 has been approved by reviewers (I'd like to take a look, but it shouldn't block the work on the ticket), I think we should notify the stakeholders that work on acceptance testing, exploratory testing, and documentation can start.

Once you have the PR deployed to Cloud for testing, please ping @approksiu, @ARWNightingale, and @vgomez-el in this thread and share credentials with them.

Please also open a documentation ticket for related_integrations similar to elastic/security-docs#4917 and elastic/security-docs#5029 and ping @joepeeples here in this thread.

It would be great if you could also try to align with the stakeholders on timing and plan a release date in Serverless.

[maximpn]

@approksiu, @ARWNightingale, and @vgomez-el could you perform testing of #178295?

• ESS environment. Use default credentials or reach me in Slack if have any troubles.
• Serverless environment. Credentials: https://p.elstc.co/paste/wEkkdXGx#9NCz27Z6HV1NfuNbkOvKn+5fLrauQCZM6SLEWD3DNAd.

@banderror there is a problem with automatic cloud deploy via tags. I had to use another approach to deploy my PR. It's only possible to deploy in ESS env atm.

UPDATE: Serverless cloud deploy works well. I updated the information above.

UPDATE 2: It seems Serverless cloud deploy stoped working.

[approksiu]

Great work Maxim, looks good to me!

[ARWNightingale]

Looks good to me!!
Just a note for future enhancements maybe we should look at disabling the add + option when we have an empty input so the user can not add more than 1 empty inputs, but this is across all similar interactions elsewhere too.

[vgomez-el]

@approksiu @banderror According to the issue title, user Story and @maximpn's PR description, I understand that this feature is only available for custom rules, but If I select another rule type rather than Custom rules, the related integrations option is displayed as well:
New Terms:

Indicator Match:

ES|QL:

Is that intended? In case it is, it should be reflected on any of the existing issues, because the current issues description can lead to confusion.

[maximpn]

Hi @vgomez-el!

Sorry for the confusion. Custom rules mean NOT Prebuilt rules since it's a prerequisite for Milestone 3 prebuilt rules customization. It's applicable to all rule types. Word custom is to only highlight we don't touch prebuilt rules.

[vgomez-el]

Sorry for not understanding it correctly, @maximpn. I have performed exploratory testing on your environment and the feature looks nice to me. I will perform the Kibana Upgrade scenarios and the CCS scenarios once I have the first build candidate of the 8.15 version. Please, Ping me if someone needs more details on the exploratory testing performed.

[joepeeples]

@maximpn and all, a docs PR is now open for serverless docs: https://github.com/elastic/staging-serverless-security-docs/pull/337

[banderror]

The feature went live in Serverless today 🚀 Thanks to everyone for your efforts!