[Security Solution] Implement normalization on read for rule_source and immutable fields

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

As part of our migration strategy to our new schema, we need to implement normalization on read for all endpoints:

kibana/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md

Lines 426 to 454 in 07706fd

	### Migration strategy
	Our migration strategy will consist of two distinct types of migration: a **migration on write** that will update the SO on Elasticsearch, and a **normalization on read**, which will transform legacy rules to the new schema **in memory** on read operations, before returning it as a response from an API endpoint.
	| API endpoint | Normalization-on-read | Migration-on-write | Comments |
	|-|-|-|-|
	| **Find Rules** - `GET /rules/_find` | <center>✅</center> | <center>❌</center> | |
	| **Read Rule** - `GET /rules` | <center>✅</center> | <center>❌</center> | |
	| **Delete Rules** - `DELETE /rules` | <center>✅</center> | <center>❌</center> | |
	| **Export Rules** - `POST /rules/_export` | <center>✅</center> | <center>❌</center> | - Endpoints is unused in UI, still used via public API <br> - See section [Exporting rules](#exporting-rules) |
	| **Import Rules** - `POST /rules/_import` | <center>✅</center> | <center>✅</center> | - See section [Importing rules](#importing-rules) |
	| **Update Rule** - `PUT /rules`| <center>✅</center> | <center>✅</center> | - Used in the UI when updating/modifying a single rule via the Rule Editing page |
	| **Patch Rule** - `PATCH /rules`| <center>✅</center> | <center>✅</center> | - Used in the UI for attaching shared exceptions list to rules |
	| **Bulk Update Rules** - `PUT /rules/_bulk_update`| <center>✅</center> | <center>✅</center> | - Deprecated and unused by the UI. Might still be used by API users |
	| **Bulk Patch Rules** - `PATCH /rules/_bulk_update`| <center>✅</center> | <center>✅</center> | - Deprecated and unused by the UI. Might still be used by API users |
	| **Perform Rule Upgrade** - `POST /prebuilt_rules/upgrade/_perform` (Internal) | <center>✅</center> | <center>✅</center> | - Current way of upgrading a prebuilt rule |
	| **(LEGACY) Install Prebuilt Rules And Timelines** - `PUT /rules/prepackaged` | <center>✅</center> | <center>✅</center> | - Legacy endpoint for installing prebuilt rules and updating rules. |
	|**Bulk Actions** - `POST /rules/_bulk_action`: | | | This endpoint includes a `dry_run` mode that is executed to evaluate preconditions and warn the user before executing the actual request. No migration logic should take place for dry run requests, i.e when `dry_run=true`, since we never write to ES when this parameter is set to `true`.|
	| <li>_**Enable and disable action**_</li> | <center>✅</center> | <center>❌</center> | - Migration-on-write is technically possible but we have decided to avoid implementing the additional logic in the AF side. Also, logic should be [migrated](https://github.com/elastic/kibana/issues/177634) soon. |
	| <li>_**Delete action**_</li> | <center>✅</center> | <center>❌</center> | - Deletes ES object but returns deleted rules data, so so normalization-on-read is enough. |
	| <li>_**Export action**_</li> | <center>✅</center> | <center>❌</center> | - See section [Exporting rules](#exporting-rules) |
	| <li>_**Duplicate rule action**_</li> | <center>✅</center> | <center>❌</center> | - Per definition, all duplicated rules will be `custom` rules. That means that all duplicated rules (the duplicates) are newly created and should get a `rule_source` of type `internal`, and no migration-on-write is neccessary. |
	| <li>_**Edit rule action**_</li> | <center>✅</center> | <center>✅</center> | - We can take advantage of the `ruleParamsModifier` to carry out the migration-on-write, regardless of the type of edit that is being performed. <br>- See implementation details in the [Bulk editing rules](#bulk-editing-rules) section.
	| **Review Rule Installation** - `POST /prebuilt_rules/installation/_review`| <center>✅</center> | <center>❌</center> | |
	| **Perform Rule Installation** - `POST /prebuilt_rules/installation/_install`| <center>✅</center> | <center>❌</center> | - Newly installed rules will be installed with new schema; but no migration-on-write should happen in this endpoint. |
	| **Review Rule Upgrade** - `POST /prebuilt_rules/upgrade/_review` | <center>✅</center> | <center>❌</center> | |
	| **Perform Rule Upgrade** - `POST /prebuilt_rules/upgrade/_perform`| <center>✅</center> | <center>✅</center> | |

• All endpoints that respond with a rule Saved Object, typed as RuleResponse, will perform normalization on read, which will transform legacy rules to the new schema in memory on read operation, before returning it as a response from an API endpoint.
• The normalization on read should be encapsulated into the new RulesManagementClient abstraction to be created in [Security Solution] RulesManagementClient refactoring. Part 1 #180128

Background

kibana/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md

Lines 457 to 538 in 07706fd

	#### Normalization on read
	All endpoints that respond with a rule Saved Object, typed as `RuleResponse`, will perform **normalization on read**, which will transform legacy rules to the new schema **in memory** on read operation, before returning it as a response from an API endpoint.
	This means that the endpoints will always respond with the rules with the new schema, while the actual rule saved object might still be stored with the legacy schema in Elasticsearch, if it still has not been migrated-on-write.
	The **normalization on read** will be carried out by a new `normalizeRuleSourceSchemaOnRuleRead` normalization function. The `internalRuleToAPIResponse` method, which is used in our endpoints to convert a rule saved object as is stored in Elasticsearch to the `RuleResponse` type which is returned to the client, calls the `commonParamsCamelToSnake` methods to convert rule parameters that are common to all rule types to what's expected in `RuleResponse`.
	Inside this method, we will use `normalizeRuleSourceSchemaOnRuleRead` to calculate the normalized values of `rule_source` and `immutable`.
	_Source: [x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.ts](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.ts)_
	```ts
	export const internalRuleToAPIResponse = (rule) => {
	return {
	...commonParamsCamelToSnake(rule.params), // <--- rule params are converted here
	...typeSpecificCamelToSnake(rule.params),
	// [... more object properties ...]
	};
	};
	export const commonParamsCamelToSnake = (params: BaseRuleParams) => {
	const { immutable, rule_source } = normalizeRuleSourceSchemaOnRuleRead(params);
	return {
	immutable,
	rule_source,
	// [... more object properties ...]
	};
	};
	```
	And the `normalizeRuleSourceSchemaOnRuleRead` can be defined so:
	_Source: x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/prebuilt_rule_schema_migrations.ts_ (New file)
	```ts
	interface OnReadNormalizationResult {
	immutable: IsRuleImmutable;
	rule_source: RuleSource
	}
	const getRuleSourceValueForRuleRead = (ruleParams: BaseRuleParams): RuleSource | undefined => {
	if (ruleParams.rule_source) {
	return ruleParams.rule_source;
	}
	if (ruleParams.immutable) {
	return {
	type: 'external'
	isCustomized: false,
	};
	}
	return {
	type: 'internal'
	};
	};
	export const normalizePrebuiltSchemaOnRuleRead = (
	ruleParams: BaseRuleParams
	): OnReadNormalizationResult => {
	/**
	* To calculate `immutable`:
	* - Checks if the `rule_source` field exists in the rule's parameters.
	* - If `rule_source` exists and its type is external, sets `immutable` to `true`. (Use case of Rules that have already been migrated-on-write)
	* - If `rule_source` does not exist, return the value of the params' `immutable` field. (Use case of Rules that have not yet been migrated on write.)
	*/
	const isRuleSourceExternal = ruleParams.rule_source && ruleParams.rule_source.type === 'external';
	const immutable = Boolean(isRuleSourceExternal) || Boolean(ruleParams.immutable);
	const rule_source = getRuleSourceValueForRuleRead(ruleParams);
	return {
	immutable,
	rule_source,
	};
	};
	```
	#### Migration on write

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)