[Ingest Manager] Rate limit agent config update

[nchaulet]

Summary

Resolve #70250

Currently in a scenario with a lot of agents, updating an agent config cause Kibana to be not responsive.

Rate limit the creation of new agent config action to not block Kibana.

The rate limit is configurable using fleet.agentConfigRollupRateLimitIntervalMs and fleet.agentConfigRollupRateLimitRequestPerInterval config variables.

Tests

Locally I enrolled 2000 agents and triggered a config change, it's still causing a spike in the event loop, but Kibana is usable.

Before

Kibana was not responsive (not able to switch between kibana)

After

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jfsiii]

LGTM, but I'm not experienced with rxjs specifically.

I think the improvements are reason enough to merge. We can benchmark, and improve further later.

[nchaulet]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 8b9a8d0

Build metrics

Saved Objects .kibana field count

id	value	diff	baseline
action	5	+5	-
action_task_params	3	+3	-
alert	22	+22	-
apm-indices	8	+8	-
apm-telemetry	238	+238	-
application_usage_transactional	2	+2	-
canvas-element	8	+8	-
canvas-workpad	5	+5	-
canvas-workpad-template	8	+8	-
cases	32	+32	-
cases-comments	17	+17	-
cases-configure	14	+14	-
cases-user-actions	10	+10	-
config	2	+2	-
dashboard	17	+17	-
endpoint:user-artifact	10	+10	-
endpoint:user-artifact-manifest	3	+3	-
epm-packages	8	+8	-
exception-list	35	+35	-
exception-list-agnostic	35	+35	-
file-upload-telemetry	2	+2	-
fleet-agent-actions	6	+6	-
fleet-agent-events	11	+11	-
fleet-agents	23	+23	-
fleet-enrollment-api-keys	10	+10	-
graph-workspace	9	+9	-
index-pattern	10	+10	-
infrastructure-ui-source	21	+21	-
ingest_manager_settings	5	+5	-
ingest-agent-configs	11	+11	-
ingest-outputs	10	+10	-
ingest-package-configs	35	+35	-
inventory-view	40	+40	-
kql-telemetry	3	+3	-
lens	7	+7	-
lens-ui-telemetry	5	+5	-
map	7	+7	-
metrics-explorer-view	22	+22	-
ml-telemetry	3	+3	-
namespace	1	+1	-
namespaces	1	+1	-
query	6	+6	-
references	4	+4	-
sample-data-telemetry	3	+3	-
search	9	+9	-
siem-detection-engine-rule-actions	8	+8	-
siem-detection-engine-rule-status	12	+12	-
siem-ui-timeline	86	+86	-
siem-ui-timeline-note	8	+8	-
siem-ui-timeline-pinned-event	7	+7	-
space	9	+9	-
telemetry	9	+9	-
timelion-sheet	13	+13	-
tsvb-validation-telemetry	2	+2	-
type	1	+1	-
ui-metric	2	+2	-
updated_at	1	+1	-
upgrade-assistant-reindex-operation	15	+15	-
upgrade-assistant-telemetry	13	+13	-
uptime-dynamic-settings	4	+4	-
url	6	+6	-
visualization	9	+9	-
total	-	+951	-

History

• 💔 Build #59391 failed 1cf2ad6
• 💚 Build #59144 succeeded b4272ad
• 💚 Build #59146 succeeded 42cc283

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[roncohen]

Excited to try this out!
I think the config should be roll*out* and not "rollup"?

I see have both agentConfigRollupRateLimitRequestPerInterval and agentConfigRollupRateLimitIntervalMs.

I think the user doesn't mind too much what the agentConfigRollupRateLimitIntervalMs is. They'd want to set a limit per second, like "50/s". I suggest we hard code the interval to 1s and let the user set a simple frequency: "agentConfigRolloutRateLimit" or "agentConfigRolloutRateLimitPerSec"