[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation

[banderror]

Addresses: #106421, #106428, #102089, #106433

Summary

This PR focuses on consolidation of indexing implementations in rule_registry (#101016). It addresses some of the sub-tasks of the parent ticket.

• Encapsulate index bootstrapping logic in a new improved API exposed by RuleDataService.
• Enforce allowed values for the datasetSuffix on the API level.
• Migrate plugins using the existing RuleDataService API to the improved one.
• Make sure index names comply with design architecture.
  • [RAC] Make sure index names comply with design architecture #102089
• Improve the API of RuleDataClient.
• Enhance index bootstrapping: support custom ILM policy per index ({registrationContext}.{datasetSuffix}).
• Enhance index bootstrapping: create index template per namespace and support rollovers properly
  • based on [Draft][Rule Registry] Create index template per namespace #107700
• Enhance index bootstrapping: support secondary aliases
  • based on [Draft][Rule Registry] Create index template per namespace #107700
• Remove EventLogService implementation
  • [RAC] Remove EventLogService implementation from rule_registry #106433

This will be addressed in follow-up PRs:

• Enhance index bootstrapping: implement suggestions for backwards compatibility (naming scheme for alias and backing indices; versioning).
• Enhance index bootstrapping: implement upgrades of existing index templates.
• Make index bootstrapping logic more robust. This is partially addressed in this PR, but more improvements are needed.
• Change the way index prefix works. UPD: [RAC] Disable RAC multi-tenancy #108506
• Add support for optional TS schema (static typing).
• Update README in rule_registry.

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[banderror]

Proper versioning according to the recent discussions will be implemented in a follow-up PR.

[banderror]

This replaces the previous implementation:

    const ruleDataClient = plugins.ruleRegistry.ruleDataService.getRuleDataClient(
      'observability',
      plugins.ruleRegistry.ruleDataService.getFullAssetName(),
      () => Promise.resolve()
    );

Seems like it's a stub. Should I keep it? Why it's needed?

[weltenwort]

The observability plugin apparently needs a ruleDataClient for use in some route handlers without intending to write any alerts itself. Does this initializeIndex create any assets which the previous implementation didn't?

[banderror]

Does this initializeIndex create any assets which the previous implementation didn't?

It will see if it should bootstrap any resources for .alerts-observability. Specifically:

• ILM policy if it's provided. In this case, it's not.
• Component templates if they are provided. In this case, they're not.

And then it will update mappings of all concrete indices matching .alerts-observability-*. This pattern shouldn't match any indices, I guess, because you're saying this ruleDataClient is not used for writing alerts? Other indices containing alerts are named like .alerts-observability.apm-* etc.

Is this ruleDataClient used for reading all the observability alerts - from .alerts-observability.apm-*, .alerts-observability.logs-* and other indices?

[marshallmain]

It seems for this use case it would be useful to expose some kind of getRuleDataReader API in RuleDataPluginService that skips all the index management steps and just returns an IRuleDataReader. If I understand correctly the registrationContext would be observability.*.

[banderror]

I was also thinking about that

[banderror]

I created a ticket for that: #111173

[marshallmain]

I went through and did a first pass of the changes here. I found a few ways that I think we can simplify the implementation further, especially regarding signalling when asynchronous template installation functions are complete and the next operations can proceed.

But, very exciting to see the new usage of ruleDataService.initializeIndex! It's a huge improvement to move the initialization and signalling logic into a single place rather than having each plugin implement it separately.

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 077ca24
• Storybooks Preview

Metrics [docs]

Unknown metric groups

API count

id	before	after	diff
ruleRegistry	95	114	+19

API count missing comments

id	before	after	diff
ruleRegistry	95	94	-1

Non-exported public API item count

id	before	after	diff
ruleRegistry	11	4	-7

History

• 💚 Build #145516 succeeded 077ca24
• 💚 Build #145302 succeeded adfe31038620ac11007891c7d33a3daa1655a398
• 💛 Build #145172 was flaky adfe31038620ac11007891c7d33a3daa1655a398
• 💚 Build #145027 succeeded 05dc4fa
• 💚 Build #144998 succeeded 495440360807431cfc2e43f4fca2a0e37386f80b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @banderror