[Security Solution] Allow users to edit related_integrations field fo…

…r custom rules (#178295)

**Resolves: #173595

## Summary

This PR adds an ability to add and edit custom rule's related integrations. Functionality is necessary to start working on [Prebuilt Rule Customization Epic Milestone 3](#174168).

## Details

Rule's related integrations represent optional dependencies on [Elastic integrations](https://docs.elastic.co/en/integrations) to ingest data. Currently prebuilt rule's related integrations are shown on rule details page. This information contains integration's name, installation status and a version mismatch warning when related integration's version dependency doesn't match with an installed integration's version. A subset of [Semver](https://semver.org/) is used to specify version dependency. Elastic prebuilt rules use only caret syntax like `^1.2.3`.

To make it possible to add and edit related integrations for custom rules the following has been done

- New internal endpoint `/internal/detection_engine/fleet/integrations/all` has been added. It returns the full list of available integrations containing title, latest available version and installed version if available. This is necessary to display an options list where users can pick a desired integration. Since some Elastic Prebuilt rules depend not only on integrations from `security` category this endpoint returns all available integrations (not only related to Security Solution).
- Rule create form has been adjusted by adding `Related Integrations` form controls
- Rule edit form has been adjusted by adding `Related Integrations` form controls
- Related integrations installation status has been adjusted to conform with the design
- Functional Jest tests have been added
- Functional tests have been added to make sure it's possible to (bulk) `create`/`patch`/`update`/`export`/`import` with related integrations
- A limited number of Cypress tests have been added

### Integration installation status

Integration installation status has been adjusted. There are following statuses shown

- `Enabled` for installed and enabled integrations. Enabled integrations are detected by checking Elastic Agent policies for presence of such an integration. It's not guaranteed the policy is picked by agents and data is being ingested.
- `Disabled` for installed and disabled integrations. An agent policy containing such an integration isn't found.
- `Not installed` for  not installed integrations.
- Nothing is shown for unknown integrations. If there is no such integration found in `/internal/detection_engine/fleet/integrations/all` result it's considered as unknown.

### Version dependency

[Semver](https://semver.org/) allows a wide range of version range declaration. Such flexibility will complicate constructing of an integration link on rule details page. Since Elastic Prebuilt rules use only caret version dependency like `^1.2.3` related integration's version dependency is limited to a subset of semver semantic. The following is supported

- A plain version dependency e.g. `1.2.3`
- Tilde version dependency  e.g. `~1.2.3`
- Caret version dependency e.g. `^1.2.3`

### Misc

- #152408 has been fixed by this PR.
- `/internal/detection_engine/fleet/integrations/installed` endpoint hasn't been removed. We need to make sure it's not needed anymore.
- E2e testing of the current functionality is complicated by dependency on EPR and difficulties to mock it. EPR periodically may respond with an error resulting in flaky Cypress tests.

### Flaky test runner results

- 🟢  [Create rule](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5632) (100 runs ESS and 100 runs in Serverless)
- 🟢  [Rule Management related integrations](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5632) (100 runs ESS and 100 runs in Serverless)

### Screenshots

![Screenshot 2024-04-16 at 10 01 25](https://github.com/elastic/kibana/assets/3775283/f41574cb-c806-4e49-97bf-9b27bf4c0f39)

![Screenshot 2024-04-16 at 10 02 03](https://github.com/elastic/kibana/assets/3775283/cf15580e-169f-4823-a579-257509c806a4)

![Screenshot 2024-04-16 at 10 02 16](https://github.com/elastic/kibana/assets/3775283/03a21eea-1014-484f-b1d2-3db81c46b8ef)

![Screenshot 2024-04-16 at 10 04 19](https://github.com/elastic/kibana/assets/3775283/06385ef4-458f-4562-bb8f-d98db9bb1fe3)

![Screenshot 2024-04-16 at 10 02 33](https://github.com/elastic/kibana/assets/3775283/edec85bf-d020-4afb-a999-4eb21255c3b6)

![Screenshot 2024-04-16 at 10 04 40](https://github.com/elastic/kibana/assets/3775283/a21c55a8-9947-44b0-ba1f-6624cd410d3e)

![Screenshot 2024-04-16 at 10 05 03](https://github.com/elastic/kibana/assets/3775283/05928a15-961b-4f67-9968-d2a32ceb86dc)

packages/kbn-doc-links/src/get_doc_links.ts

@@ -480,6 +480,7 @@ export const getDocLinks = ({ kibanaBranch, buildFlavor }: GetDocLinkOptions): D
       privileges: `${SECURITY_SOLUTION_DOCS}endpoint-management-req.html`,
       manageDetectionRules: `${SECURITY_SOLUTION_DOCS}rules-ui-management.html`,
       createEsqlRuleType: `${SECURITY_SOLUTION_DOCS}rules-ui-create.html#create-esql-rule`,
+      ruleUiAdvancedParams: `${SECURITY_SOLUTION_DOCS}rules-ui-create.html#rule-ui-advanced-params`,
       entityAnalytics: {
         riskScorePrerequisites: `${SECURITY_SOLUTION_DOCS}ers-requirements.html`,
         hostRiskScore: `${SECURITY_SOLUTION_DOCS}host-risk-score.html`,

packages/kbn-doc-links/src/types.ts

@@ -355,6 +355,7 @@ export interface DocLinks {
     readonly privileges: string;
     readonly manageDetectionRules: string;
     readonly createEsqlRuleType: string;
+    readonly ruleUiAdvancedParams: string;
     readonly entityAnalytics: {
       readonly riskScorePrerequisites: string;
       readonly hostRiskScore: string;

x-pack/plugins/security_solution/common/api/detection_engine/fleet_integrations/get_all_integrations/get_all_integrations_route.ts

@@ -0,0 +1,12 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { Integration } from '../model/integrations';
+
+export interface GetAllIntegrationsResponse {
+  integrations: Integration[];
+}

x-pack/plugins/security_solution/common/api/detection_engine/fleet_integrations/index.ts

@@ -5,7 +5,10 @@
  * 2.0.
  */
 
+export * from './get_all_integrations/get_all_integrations_route';
+
 export * from './get_installed_integrations/get_installed_integrations_route';
 export * from './urls';
 
+export * from './model/integrations';
 export * from './model/installed_integrations';

x-pack/plugins/security_solution/common/api/detection_engine/fleet_integrations/model/integrations.ts

@@ -0,0 +1,101 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+// -------------------------------------------------------------------------------------------------
+// Fleet Package Integration
+
+/**
+ * Information about a Fleet integration including info about its package.
+ *
+ * @example
+ * {
+ *   package_name: 'aws',
+ *   package_title: 'AWS',
+ *   integration_name: 'cloudtrail',
+ *   integration_title: 'AWS CloudTrail',
+ *   latest_package_version: '1.2.3',
+ *   is_installed: false
+ *   is_enabled: false
+ * }
+ *
+ * @example
+ * {
+ *   package_name: 'aws',
+ *   package_title: 'AWS',
+ *   integration_name: 'cloudtrail',
+ *   integration_title: 'AWS CloudTrail',
+ *   latest_package_version: '1.16.1',
+ *   installed_package_version: '1.16.1',
+ *   is_installed: true
+ *   is_enabled: false
+ * }
+ *
+ * @example
+ * {
+ *   package_name: 'system',
+ *   package_title: 'System',
+ *   latest_package_version: '2.0.1',
+ *   installed_package_version: '1.13.0',
+ *   is_installed: true
+ *   is_enabled: true
+ * }
+ *
+ */
+export interface Integration {
+  /**
+   * Name is a unique package id within a given cluster.
+   * There can't be 2 or more different packages with the same name.
+   * @example 'aws'
+   */
+  package_name: string;
+
+  /**
+   * Title is a user-friendly name of the package that we show in the UI.
+   * @example 'AWS'
+   */
+  package_title: string;
+
+  /**
+   * Whether the package is installed
+   */
+  is_installed: boolean;
+
+  /**
+   * Whether this integration is enabled
+   */
+  is_enabled: boolean;
+
+  /**
+   * Version of the latest available package. Semver-compatible.
+   * @example '1.2.3'
+   */
+  latest_package_version: string;
+
+  /**
+   * Version of the installed package. Semver-compatible.
+   * @example '1.2.3'
+   */
+  installed_package_version?: string;
+
+  /**
+   * Name identifies an integration within its package.
+   * Undefined when package name === integration name. This indicates that it's the only integration
+   * within this package.
+   * @example 'cloudtrail'
+   * @example undefined
+   */
+  integration_name?: string;
+
+  /**
+   * Title is a user-friendly name of the integration that we show in the UI.
+   * Undefined when package name === integration name. This indicates that it's the only integration
+   * within this package.
+   * @example 'AWS CloudTrail'
+   * @example undefined
+   */
+  integration_title?: string;
+}

x-pack/plugins/security_solution/common/api/detection_engine/fleet_integrations/urls.ts

@@ -7,5 +7,7 @@
 
 import { INTERNAL_DETECTION_ENGINE_URL as INTERNAL_URL } from '../../../constants';
 
+export const GET_ALL_INTEGRATIONS_URL = `${INTERNAL_URL}/fleet/integrations/all` as const;
+
 export const GET_INSTALLED_INTEGRATIONS_URL =
   `${INTERNAL_URL}/fleet/integrations/installed` as const;

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts

@@ -53,11 +53,11 @@ import {
   MaxSignals,
   ThreatArray,
   SetupGuide,
+  RelatedIntegrationArray,
   RuleObjectId,
   RuleSignatureId,
   IsRuleImmutable,
   RuleSource,
-  RelatedIntegrationArray,
   RequiredFieldArray,
   RuleQuery,
   IndexPatternArray,
@@ -136,6 +136,7 @@ export const BaseDefaultableFields = z.object({
   max_signals: MaxSignals.optional(),
   threat: ThreatArray.optional(),
   setup: SetupGuide.optional(),
+  related_integrations: RelatedIntegrationArray.optional(),
 });
 
 export type BaseCreateProps = z.infer<typeof BaseCreateProps>;
@@ -163,7 +164,6 @@ export const ResponseFields = z.object({
   created_at: z.string().datetime(),
   created_by: z.string(),
   revision: z.number().int().min(0),
-  related_integrations: RelatedIntegrationArray,
   required_fields: RequiredFieldArray,
   execution_summary: RuleExecutionSummary.optional(),
 });

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.schema.yaml

@@ -131,6 +131,9 @@ components:
         setup:
           $ref: './common_attributes.schema.yaml#/components/schemas/SetupGuide'
 
+        related_integrations:
+          $ref: './common_attributes.schema.yaml#/components/schemas/RelatedIntegrationArray'
+
     BaseCreateProps:
       x-inline: true
       allOf:
@@ -178,13 +181,11 @@ components:
         revision:
           type: integer
           minimum: 0
-        # NOTE: For now, Related Integrations and Required Fields are
+        # NOTE: For now, Required Fields are
         # supported for prebuilt rules only. We don't want to allow users to edit these 3
         # fields via the API. If we added them to baseParams.defaultable, they would
         # become a part of the request schema as optional fields. This is why we add them
         # here, in order to add them only to the response schema.
-        related_integrations:
-          $ref: './common_attributes.schema.yaml#/components/schemas/RelatedIntegrationArray'
         required_fields:
           $ref: './common_attributes.schema.yaml#/components/schemas/RequiredFieldArray'
         execution_summary:

x-pack/plugins/security_solution/public/common/mock/create_react_query_wrapper.tsx

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React from 'react';
+import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
+
+export function createReactQueryWrapper(): React.FC {
+  const queryClient = new QueryClient({
+    defaultOptions: {
+      queries: {
+        // Turn retries off, otherwise we won't be able to test errors
+        retry: false,
+      },
+    },
+  });
+
+  // eslint-disable-next-line react/display-name
+  return ({ children }) => (
+    <QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
+  );
+}

x-pack/plugins/security_solution/public/common/mock/index.ts

@@ -19,3 +19,4 @@ export * from './test_providers';
 export * from './timeline_results';
 export * from './utils';
 export * from './create_store';
+export * from './create_react_query_wrapper';

x-pack/plugins/security_solution/public/detection_engine/fleet_integrations/api/__mocks__/api_client.ts

@@ -5,13 +5,49 @@
  * 2.0.
  */
 
-import type { GetInstalledIntegrationsResponse } from '../../../../../common/api/detection_engine/fleet_integrations';
 import type {
+  GetAllIntegrationsResponse,
+  GetInstalledIntegrationsResponse,
+} from '../../../../../common/api/detection_engine/fleet_integrations';
+import type {
+  FetchAllIntegrationsArgs,
   FetchInstalledIntegrationsArgs,
   IFleetIntegrationsApiClient,
 } from '../api_client_interface';
 
 export const fleetIntegrationsApi: jest.Mocked<IFleetIntegrationsApiClient> = {
+  fetchAllIntegrations: jest
+    .fn<Promise<GetAllIntegrationsResponse>, [FetchAllIntegrationsArgs]>()
+    .mockResolvedValue({
+      integrations: [
+        {
+          package_name: 'o365',
+          package_title: 'Microsoft 365',
+          latest_package_version: '1.2.0',
+          installed_package_version: '1.0.0',
+          is_installed: false,
+          is_enabled: false,
+        },
+        {
+          package_name: 'atlassian_bitbucket',
+          package_title: 'Atlassian Bitbucket',
+          latest_package_version: '1.0.1',
+          installed_package_version: '1.0.1',
+          integration_name: 'audit',
+          integration_title: 'Audit Logs',
+          is_installed: true,
+          is_enabled: true,
+        },
+        {
+          package_name: 'system',
+          package_title: 'System',
+          latest_package_version: '1.6.4',
+          installed_package_version: '1.6.4',
+          is_installed: true,
+          is_enabled: true,
+        },
+      ],
+    }),
   fetchInstalledIntegrations: jest
     .fn<Promise<GetInstalledIntegrationsResponse>, [FetchInstalledIntegrationsArgs]>()
     .mockResolvedValue({

x-pack/plugins/security_solution/public/detection_engine/fleet_integrations/api/api_client.ts

@@ -5,16 +5,31 @@
  * 2.0.
  */
 
-import type { GetInstalledIntegrationsResponse } from '../../../../common/api/detection_engine/fleet_integrations';
-import { GET_INSTALLED_INTEGRATIONS_URL } from '../../../../common/api/detection_engine/fleet_integrations';
+import type {
+  GetAllIntegrationsResponse,
+  GetInstalledIntegrationsResponse,
+} from '../../../../common/api/detection_engine/fleet_integrations';
+import {
+  GET_ALL_INTEGRATIONS_URL,
+  GET_INSTALLED_INTEGRATIONS_URL,
+} from '../../../../common/api/detection_engine/fleet_integrations';
 import { KibanaServices } from '../../../common/lib/kibana';
 
 import type {
+  FetchAllIntegrationsArgs,
   FetchInstalledIntegrationsArgs,
   IFleetIntegrationsApiClient,
 } from './api_client_interface';
 
 export const fleetIntegrationsApi: IFleetIntegrationsApiClient = {
+  fetchAllIntegrations: (args: FetchAllIntegrationsArgs): Promise<GetAllIntegrationsResponse> => {
+    return http().fetch<GetAllIntegrationsResponse>(GET_ALL_INTEGRATIONS_URL, {
+      method: 'GET',
+      version: '1',
+      signal: args.signal,
+    });
+  },
+
   fetchInstalledIntegrations: (
     args: FetchInstalledIntegrationsArgs
   ): Promise<GetInstalledIntegrationsResponse> => {

x-pack/plugins/security_solution/public/detection_engine/fleet_integrations/api/api_client_interface.ts

@@ -5,9 +5,19 @@
  * 2.0.
  */
 
-import type { GetInstalledIntegrationsResponse } from '../../../../common/api/detection_engine/fleet_integrations';
+import type {
+  GetInstalledIntegrationsResponse,
+  GetAllIntegrationsResponse,
+} from '../../../../common/api/detection_engine/fleet_integrations';
 
 export interface IFleetIntegrationsApiClient {
+  /**
+   * Fetch all integrations with installed and enabled statuses
+   *
+   * @throws An error if response is not OK
+   */
+  fetchAllIntegrations(args: FetchAllIntegrationsArgs): Promise<GetAllIntegrationsResponse>;
+
   /**
    * Fetch all installed integrations.
    * @throws An error if response is not OK
@@ -17,6 +27,13 @@ export interface IFleetIntegrationsApiClient {
   ): Promise<GetInstalledIntegrationsResponse>;
 }
 
+export interface FetchAllIntegrationsArgs {
+  /**
+   * Optional signal for cancelling the request.
+   */
+  signal?: AbortSignal;
+}
+
 export interface FetchInstalledIntegrationsArgs {
   /**
    * Array of Fleet packages to filter for.

x-pack/plugins/security_solution/public/detection_engine/rule_creation/components/related_integrations/default_related_integration.ts

@@ -0,0 +1,8 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export const DEFAULT_RELATED_INTEGRATION = { package: '', version: '' };

x-pack/plugins/security_solution/public/detection_engine/rule_creation/components/related_integrations/index.ts

@@ -0,0 +1,8 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export { RelatedIntegrations } from './related_integrations';