[Security Solution] Extend upgrade perform endpoint logic (#191439)

Fixes: #166376 (main ticket)
Fixes: #186544 (handling of
specific fields)
Fixes: #180195 (replace PATCH
with PUT logic on rule upgrade)

## Summary

- Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way
that works with prebuilt rules customized by users and resolve conflicts
between user customizations and updates from Elastic.
- Handles special fields under the hood (see below)
- Replaces the update prebuilt rule logic to work with PUT instead of
PATCH.

### Rough implementation plan
- For each `upgradeableRule`, we attempt to build the payload necessary
to pass to `upgradePrebuiltRules()`, which is of type
`PrebuiltRuleAsset`. So we retrieve the field names from
`FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them.
- If any of those `field`s are non-upgreadable, (i.e. its value needs to
be handled under the hood) we do so in `determineFieldUpgradeStatus`.
- Otherwise, we continue to build a `FieldUpgradeSpecifier` for each
field, which will help us determine if that field needs to be set to the
base, current, target version, OR if it needs to be calculated as a
MERGED value, or it is passed in the request payload as a RESOLVED
value.
- Notice that we are iterating over "flat" (non-grouped) fields which
are part of the `PrebuiltRuleAsset` schema. This means that mapping is
necessary between these flat fields and the diffable (grouped) fields
that are used in the API contract, part of `DiffableRule`. For example,
if we try to determine the value for the `query` field, we will need to
look up for its value in the `eql_query` field if the target rule is
`eql` or in `esql_query` if the target rule is `esql`. All these
mappings can be found in `diffable_rule_fields_mappings.ts`.
- Once a `FieldUpgradeSpecifier` has been retrieved for each field of
the payload we are building, retrieve its actual value: either fetching
it from the base, current or target versions of the rule, from the three
way diff calculation, or retrieving it from the request payload if it
resolved.
- Do this for all upgreadable rules, and the pass the payload array into
`upgradePrebuiltRules()`.
- **IMPORTANT:** The upgrade prebuilt rules logic has been changed from
PATCH to PUT. That means that if the next version of a rule removes a
field, and the user updates to that target version, those fields will be
undefined in the resulting rule. **Additional example:** a installs a
rule, and creates a `timeline_id` for it rule by modifying it. If
neither the next version (target version) still does not have a
`timeline_id` field for it, and the user updates to that target version
fully (without resolving the conflict), that field will not exist
anymore in the resulting rule.

## Acceptance criteria

- [x] Extend the contract of the API endpoint according to the
[POC](#144060):
- [x] Add the ability to pick the `MERGED` version for rule upgrades. If
the `MERGED` version is selected, the diffs are recalculated and the
rule fields are updated to the result of the diff calculation. This is
only possible if all field diffs return a `conflict` value of either
`NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`,
reject the request with an error specifying that there are conflicts,
and that they must be resolved on a per-field basis.
- [x] Calculate diffs inside this endpoint, when the value of
`pick_version` is `MERGED`.
- [x] Add the ability to specify rule field versions, to update specific
fields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |
'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in
[PoC](#144060) for details)

## Handling of special fields

Specific fields are handled under the hood based on
#186544

See implementation in
`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,
which imports fields to handle under the hood:
- `DiffableFieldsToOmit`
- `FieldsToUpdateToCurrentVersion`

## Edge cases

- [x] If target version of rule has a **rule type change**, check that
all `pick_version`, at all levels, match `TARGET`. Otherwise, create new
error and add to ruleErrors array.
- [x] if a rule has a specific `targetVersion.type` (for example, EQL)
and the user includes in its `fields` object of the request payload any
fields which do not match that rule type (in this case, for example,
sending in `machine_learning_job_id` as part of `fields`), throw an
error for that rule.
- [x] Calculation of field diffs: what happens if some fields have a
conflict value of `NON_SOLVABLE`:
- [x] If the whole rule is being updated to `MERGED`, and **ANY** fields
return with a `NON_SOLVABLE` conflict, reject the whole update for that
rule: create new error and add to ruleErrors array.
- [x] **EXCEPTION** for case above: the whole rule is being updated to
`MERGED`, and one or more of the fields return with a `NON_SOLVABLE`
conflict, BUT those same fields have a specific `pick_version` for them
in the `fields` object which **ARE NOT** `MERGED`. No error should be
reported in this case.
- [x] The whole rule is being updated to any `pick_version` other than
MERGED, but any specific field in the `fields` object is set to upgrade
to `MERGED`, and the diff for that fields returns a `NON_SOLVABLE`
conflict. In that case, create new error and add to ruleErrors array.

### TODO

- [[Security Solution] Add InvestigationFields and AlertSuppression
fields to the upgrade workflow
[#190597]](#190597):
InvestigationFields is already working, but AlertSuppression is still
currently handled under the hood to update to current version.


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Maxim Palenov <[email protected]>

x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/model/diff/diffable_rule/diffable_rule.ts

@@ -174,6 +174,17 @@ export const DiffableNewTermsFields = z.object({
   alert_suppression: AlertSuppression.optional(),
 });
 
+export const DiffableFieldsByTypeUnion = z.discriminatedUnion('type', [
+  DiffableCustomQueryFields,
+  DiffableSavedQueryFields,
+  DiffableEqlFields,
+  DiffableEsqlFields,
+  DiffableThreatMatchFields,
+  DiffableThresholdFields,
+  DiffableMachineLearningFields,
+  DiffableNewTermsFields,
+]);
+
 /**
  * Represents a normalized rule object that is suitable for passing to the diff algorithm.
  * Every top-level field of a diffable rule can be compared separately on its own.
@@ -200,18 +211,6 @@ export const DiffableNewTermsFields = z.object({
  * NOTE: Every top-level field in a DiffableRule MUST BE LOGICALLY INDEPENDENT from other
  * top-level fields.
  */
-
-export const DiffableFieldsByTypeUnion = z.discriminatedUnion('type', [
-  DiffableCustomQueryFields,
-  DiffableSavedQueryFields,
-  DiffableEqlFields,
-  DiffableEsqlFields,
-  DiffableThreatMatchFields,
-  DiffableThresholdFields,
-  DiffableMachineLearningFields,
-  DiffableNewTermsFields,
-]);
-
 export type DiffableRule = z.infer<typeof DiffableRule>;
 export const DiffableRule = z.intersection(DiffableCommonFields, DiffableFieldsByTypeUnion);
 
@@ -246,3 +245,22 @@ export const DiffableAllFields = DiffableCommonFields.merge(
   .merge(DiffableMachineLearningFields.omit({ type: true }))
   .merge(DiffableNewTermsFields.omit({ type: true }))
   .merge(z.object({ type: DiffableRuleTypes }));
+
+const getRuleTypeFields = (schema: z.ZodObject<z.ZodRawShape>): string[] =>
+  Object.keys(schema.shape);
+
+const createDiffableFieldsPerRuleType = (specificFields: z.ZodObject<z.ZodRawShape>): string[] => [
+  ...getRuleTypeFields(DiffableCommonFields),
+  ...getRuleTypeFields(specificFields),
+];
+
+export const DIFFABLE_RULE_TYPE_FIELDS_MAP = new Map<DiffableRuleTypes, string[]>([
+  ['query', createDiffableFieldsPerRuleType(DiffableCustomQueryFields)],
+  ['saved_query', createDiffableFieldsPerRuleType(DiffableSavedQueryFields)],
+  ['eql', createDiffableFieldsPerRuleType(DiffableEqlFields)],
+  ['esql', createDiffableFieldsPerRuleType(DiffableEsqlFields)],
+  ['threat_match', createDiffableFieldsPerRuleType(DiffableThreatMatchFields)],
+  ['threshold', createDiffableFieldsPerRuleType(DiffableThresholdFields)],
+  ['machine_learning', createDiffableFieldsPerRuleType(DiffableMachineLearningFields)],
+  ['new_terms', createDiffableFieldsPerRuleType(DiffableNewTermsFields)],
+]);

x-pack/plugins/security_solution/common/api/detection_engine/prebuilt_rules/perform_rule_upgrade/perform_rule_upgrade_route.ts

@@ -11,30 +11,65 @@ import { RuleResponse } from '../../model/rule_schema/rule_schemas.gen';
 import { AggregatedPrebuiltRuleError, DiffableAllFields } from '../model';
 import { RuleSignatureId, RuleVersion } from '../../model';
 
+export type Mode = z.infer<typeof Mode>;
+export const Mode = z.enum(['ALL_RULES', 'SPECIFIC_RULES']);
+export type ModeEnum = typeof Mode.enum;
+export const ModeEnum = Mode.enum;
+
 export type PickVersionValues = z.infer<typeof PickVersionValues>;
 export const PickVersionValues = z.enum(['BASE', 'CURRENT', 'TARGET', 'MERGED']);
 export type PickVersionValuesEnum = typeof PickVersionValues.enum;
 export const PickVersionValuesEnum = PickVersionValues.enum;
 
+// Specific handling of special fields according to:
+// https://github.com/elastic/kibana/issues/186544
+export const FIELDS_TO_UPGRADE_TO_CURRENT_VERSION = [
+  'enabled',
+  'exceptions_list',
+  'alert_suppression',
+  'actions',
+  'throttle',
+  'response_actions',
+  'meta',
+  'output_index',
+  'namespace',
+  'alias_purpose',
+  'alias_target_id',
+  'outcome',
+  'concurrent_searches',
+  'items_per_search',
+] as const;
+
+export const NON_UPGRADEABLE_DIFFABLE_FIELDS = [
+  'type',
+  'rule_id',
+  'version',
+  'author',
+  'license',
+] as const;
+
+type NON_UPGRADEABLE_DIFFABLE_FIELDS_TO_OMIT_TYPE = {
+  readonly [key in (typeof NON_UPGRADEABLE_DIFFABLE_FIELDS)[number]]: true;
+};
+
+// This transformation is needed to have Zod's `omit` accept the rule fields that need to be omitted
+export const DiffableFieldsToOmit = NON_UPGRADEABLE_DIFFABLE_FIELDS.reduce((acc, field) => {
+  return { ...acc, [field]: true };
+}, {} as NON_UPGRADEABLE_DIFFABLE_FIELDS_TO_OMIT_TYPE);
+
 /**
  * Fields upgradable by the /upgrade/_perform endpoint.
  * Specific fields are omitted because they are not upgradeable, and
  * handled under the hood by endpoint logic.
  * See: https://github.com/elastic/kibana/issues/186544
  */
 export type DiffableUpgradableFields = z.infer<typeof DiffableUpgradableFields>;
-export const DiffableUpgradableFields = DiffableAllFields.omit({
-  type: true,
-  rule_id: true,
-  version: true,
-  author: true,
-  license: true,
-});
+export const DiffableUpgradableFields = DiffableAllFields.omit(DiffableFieldsToOmit);
 
 export type FieldUpgradeSpecifier<T> = z.infer<
   ReturnType<typeof fieldUpgradeSpecifier<z.ZodType<T>>>
 >;
-const fieldUpgradeSpecifier = <T extends z.ZodTypeAny>(fieldSchema: T) =>
+export const fieldUpgradeSpecifier = <T extends z.ZodTypeAny>(fieldSchema: T) =>
   z.discriminatedUnion('pick_version', [
     z
       .object({

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_status/get_prebuilt_rules_status_route.ts

@@ -13,7 +13,7 @@ import { buildSiemResponse } from '../../../routes/utils';
 import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt_rule_assets_client';
 import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
 import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
-import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets';
+import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
 
 export const getPrebuiltRulesStatusRoute = (router: SecuritySolutionPluginRouter) => {
   router.versioned
@@ -44,7 +44,7 @@ export const getPrebuiltRulesStatusRoute = (router: SecuritySolutionPluginRouter
             ruleObjectsClient,
           });
           const { currentRules, installableRules, upgradeableRules, totalAvailableRules } =
-            getVersionBuckets(ruleVersionsMap);
+            getRuleGroups(ruleVersionsMap);
 
           const body: GetPrebuiltRulesStatusResponseBody = {
             stats: {

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_installation/perform_rule_installation_route.ts

@@ -25,9 +25,9 @@ import { createPrebuiltRuleAssetsClient } from '../../logic/rule_assets/prebuilt
 import { createPrebuiltRules } from '../../logic/rule_objects/create_prebuilt_rules';
 import { createPrebuiltRuleObjectsClient } from '../../logic/rule_objects/prebuilt_rule_objects_client';
 import { fetchRuleVersionsTriad } from '../../logic/rule_versions/fetch_rule_versions_triad';
-import { getVersionBuckets } from '../../model/rule_versions/get_version_buckets';
 import { performTimelinesInstallation } from '../../logic/perform_timelines_installation';
 import { PREBUILT_RULES_OPERATION_SOCKET_TIMEOUT_MS } from '../../constants';
+import { getRuleGroups } from '../../model/rule_groups/get_rule_groups';
 
 export const performRuleInstallationRoute = (router: SecuritySolutionPluginRouter) => {
   router.versioned
@@ -80,7 +80,7 @@ export const performRuleInstallationRoute = (router: SecuritySolutionPluginRoute
             ruleObjectsClient,
             versionSpecifiers: mode === 'ALL_RULES' ? undefined : request.body.rules,
           });
-          const { currentRules, installableRules } = getVersionBuckets(ruleVersionsMap);
+          const { currentRules, installableRules } = getRuleGroups(ruleVersionsMap);
 
           // Perform all the checks we can before we start the upgrade process
           if (mode === 'SPECIFIC_RULES') {

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.test.ts

@@ -0,0 +1,60 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { assertDiffableFieldsMatchRuleType } from './assert_diffable_fields_match_rule_type';
+import { DIFFABLE_RULE_TYPE_FIELDS_MAP } from '../../../../../../common/api/detection_engine';
+
+describe('assertDiffableFieldsMatchRuleType', () => {
+  describe('valid scenarios -', () => {
+    it('should validate all fields in DIFFABLE_RULE_TYPE_FIELDS_MAP', () => {
+      DIFFABLE_RULE_TYPE_FIELDS_MAP.forEach((fields, ruleType) => {
+        expect(() => {
+          assertDiffableFieldsMatchRuleType(fields, ruleType);
+        }).not.toThrow();
+      });
+    });
+
+    it('should not throw an error for valid upgradeable fields', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['name', 'description', 'severity'], 'query');
+      }).not.toThrow();
+    });
+
+    it('should handle valid rule type correctly', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['eql_query'], 'eql');
+      }).not.toThrow();
+    });
+
+    it('should handle empty upgradeable fields array', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType([], 'query');
+      }).not.toThrow();
+    });
+  });
+
+  describe('invalid scenarios -', () => {
+    it('should throw an error for invalid upgradeable fields', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['invalid_field'], 'query');
+      }).toThrow("invalid_field is not a valid upgradeable field for type 'query'");
+    });
+
+    it('should throw for incompatible rule types', () => {
+      expect(() => {
+        assertDiffableFieldsMatchRuleType(['eql_query'], 'query');
+      }).toThrow("eql_query is not a valid upgradeable field for type 'query'");
+    });
+
+    it('should throw an error for an unknown rule type', () => {
+      expect(() => {
+        // @ts-expect-error - unknown rule
+        assertDiffableFieldsMatchRuleType(['name'], 'unknown_type');
+      }).toThrow();
+    });
+  });
+});

x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/assert_diffable_fields_match_rule_type.ts

@@ -0,0 +1,40 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { DiffableRuleTypes } from '../../../../../../common/api/detection_engine';
+import { DIFFABLE_RULE_TYPE_FIELDS_MAP } from '../../../../../../common/api/detection_engine';
+
+/**
+ * Validates that the upgradeable (diffable) fields match the target rule type's diffable fields.
+ *
+ * This function is used in the rule upgrade process to ensure that the fields
+ * specified for upgrade in the request body are valid for the target rule type.
+ * It checks each upgradeable field provided in body.rule[].fields against the
+ * set of diffable fields for the target rule type.
+ *
+ * @param {string[]} diffableFields - An array of field names to be upgraded.
+ * @param {string} ruleType - A rule type (e.g., 'query', 'eql', 'machine_learning').
+ * @throws {Error} If an upgradeable field is not valid for the target rule type.
+ *
+ * @examples
+ * assertDiffableFieldsMatchRuleType(['kql_query', 'severity'], 'query');
+ * assertDiffableFieldsMatchRuleType(['esql_query', 'description'], 'esql');
+ * assertDiffableFieldsMatchRuleType(['machine_learning_job_id'], 'eql'); // throws error
+ *
+ * @see {@link DIFFABLE_RULE_TYPE_FIELDS_MAP} in diffable_rule.ts for the mapping of rule types to their diffable fields.
+ */
+export const assertDiffableFieldsMatchRuleType = (
+  diffableFields: string[],
+  ruleType: DiffableRuleTypes
+) => {
+  const diffableFieldsForType = new Set(DIFFABLE_RULE_TYPE_FIELDS_MAP.get(ruleType));
+  for (const diffableField of diffableFields) {
+    if (!diffableFieldsForType.has(diffableField)) {
+      throw new Error(`${diffableField} is not a valid upgradeable field for type '${ruleType}'`);
+    }
+  }
+};