feat: multi SCP composition

[pkoutsovasilis]

Overview

This PR implements support for multiple StackConfigPolicies (SCPs) targeting the same Elasticsearch cluster or Kibana instance, using a weight-based priority system for deterministic policy composition.

Key Features

Weight-Based Priority System

• Policies are merged in order of weight (lower weight takes precedence)
• Default weight: 0

Conflict Detection

Conflicts are detected across multiple dimensions and will prevent policy application:

Conflict Type	Condition	Result
Weight Conflict	Two or more policies with identical weights target the same Elasticsearch/Kibana	❌ Conflict
SecretMount Name Conflict	Different policies define SecretMount with same SecretName	❌ Conflict
SecretMount Path Conflict	Different policies define SecretMount with same MountPath	❌ Conflict
Different Weights	Policies have different weights and none of the above applies	✅ Pass - lower weight wins

Important: Even if two policies with the same weight have non-overlapping resources, they still conflict because the weight collision makes the merge order ambiguous.

Configuration Merging Behaviour

Different merge strategies are applied based on the configuration type:

• Deep Merge (recursive merging):

  • ClusterSettings
  • Config
  • SnapshotLifecyclePolicies
  • SecurityRoleMappings
  • IndexLifecyclePolicies
  • IngestPipelines
  • IndexTemplates.ComposableIndexTemplates
  • IndexTemplates.ComponentTemplates

• Top-Level Key Replacement (entire keys replaced):

  • SnapshotRepositories - each repository configuration is treated atomically

• Union Merge (with conflict detection):

  • SecretMounts - conflicts on duplicate SecretName OR duplicate MountPath
  • SecureSettings - merges by SecretName+Key, lower weight wins (no conflicts)

Multi-Soft-Owner Secret Management

File Settings and Policy Config Secrets:

• Now support multiple soft owners
• Secrets are only deleted when all referencing soft-owners are removed
• Uses eck.k8s.elastic.co/owner-refs annotation with JSON-encoded map of owner namespaced names

Secret Sources:

• Remain single soft owner (existing behaviour unchanged)

This prevents secret leakage while enabling proper cleanup when policies are deleted.

Related Issues

• Supersedes feat: Multi StackConfigPolicy per cluster #8793

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/api-reference/main.md

[kvalliyurnatt]

Just a general question/comment that came to my mind, do we want to have any limits on the number of SCPs that can be associated to a cluster ?
I was just trying to think what potential issues we might face when we have a large number of SCPs(if that is even a practical scenario) associated with a single ES cluster and if there is a practical maximum that we can enforce. One thing that came to my mind while thinking about scale was the annotation for soft owners, maybe we run into some kind of limit with the annotation map size? (I think the annotation limit is 256KB which I think should not be something to worry about ?)
Wondering if there are any other such things to consider.

[pebrc]

I did a first pass, just looking at the code. I have not tested it yet. Will try to find some more time later today.

[pkoutsovasilis]

buildkite test this -f p=gke,t=TestStackConfigPolicy*

[pkoutsovasilis]

buildkite test this -f p=gke,t=TestStackConfigPolicy*

[pebrc]

LGTM, nice work!

I think we have two follow-up items:

1. improve the error/change attribution

The problem mentioned by @barkbay earlier, is worse for errors that are displayed in the status resource for any contributing policy and we currently leave it up to the user to trace back from which source it came. Can you maybe raise an issue for that?

NAMESPACE        NAME                        READY   PHASE             AGE   WEIGHT
elastic-system   elasticsearch-only-policy   1/2     ApplyingChanges   12m   0
elastic-system   kibana-only-policy          1/2     ApplyingChanges   12m   9

3. documentation (needs to go into the docs-content repo)

[kvalliyurnatt]

@pkoutsovasilis wanted to follow up about my comment from earlier

#8917 (comment)

wondering if we should document what limits we might hit and maybe do some sort of scale testing to see if at a certain number it becomes problematic ? wdyt ?

[pebrc]

@pkoutsovasilis wanted to follow up about my comment from earlier

#8917 (comment)

wondering if we should document what limits we might hit and maybe do some sort of scale testing to see if at a certain number it becomes problematic ? wdyt ?

+1 to that. I tested only with a trivial number of SCPs < 10. Maybe we should see if at higher cardinalities our reconciliation algorithm runs into problems. This could inform the documentation we write as a follow up and would allow us to give some guidance to users.

Just looking at the max annotation size for the soft-owners and assuming no other annotations exist, we could support probably ~ 2000 owners (63 char ns + 63 char name ~ 130 bytes + 32 bytes for the key). While this won't likely be the limitation I don't think we should advertise this as a limit. My gut feeling is that the number of SCPs should be < 100 and the ones targeting a single ES cluster < 10. But it would be good to have more than gut feel numbers 😄

[pkoutsovasilis]

Thanks for the follow-up @kvalliyurnatt! To be honest, as @pebrc already captured in his comment above, I'm not particularly worried about the owners annotations - those numbers are way beyond what we would ever advertise. Thinking about it, the limit that's more realistic to reach, I believe, is the stack config secret limit. We can easily hit that even when merging just 2 StackConfigPolicies that are quite lengthy, which could make us exceed the 1MB size limit of a secret. Everything else, except for the stack config secret size and the owners annotation, is pretty much the same, so this PR doesn't make any other limits easier to exceed.

So to accommodate for these my proposal is to limit the maximum number of SCPs that can target a given stack component to be 10 (it feels like a fairly high number to me). Also for the config policy secret size, we could check if the error coming from here is an IsRequestEntityTooLargeError, and if yes, set the status of the StackConfigPolicy accordingly. How does that sound @kvalliyurnatt @pebrc ?

[pebrc]

I would not put a limit in code. What I was suggesting was to do a bit more "scale" testing. The overall size limit of the settings file is well known. This PR does not change anything in this regard. We are pretty confident that the annotation size is not a limitation in practice. What we do not know is how the controller behaves if there are very many SCPs that need to be reconciled on each and every change. How many conflicts will we see (if any). Any other issues we don't know of. So all I am arguing for is to test the limits of this implementation a bit before we give it to our users.

[pkoutsovasilis]

buildkite test this -f p=gke,t=TestStackConfigPolicy*

[pkoutsovasilis]

buildkite test this -f p=gke,t=TestStackConfigPolicy*

[pkoutsovasilis]

Follow-up

I've created #8945 to track the enhancement for improved error attribution in multi-SCP status reporting.

---

I ran scale testing with 100 SCPs targeting a single Elasticsearch cluster. Here are the key findings:

Test Steps:

1. Deploy 100 StackConfigPolicies targeting one ES cluster
2. Add a label change to the Elasticsearch CR to trigger reconciliation of all SCPs
3. Delete 50 SCPs
4. Delete remaining 50 SCPs

Results - Optimistic Concurrency Conflicts:
The system handled 100 SCPs gracefully. Conflicts occurred as expected with optimistic concurrency control and were resolved through automatic retries:

• Step 1 (100 SCPs deployed): 13 conflicts on secrets
• Step 2 (ES label change, full SCP reconciliation): 0 conflicts
• Step 3 (50 SCPs deleted): 45 conflicts on secrets
• Step 4 (all SCPs deleted): 38 conflicts on secrets

All conflicts were transient "object has been modified" errors that resolved via retry. The annotation size limit (256KB) was not an issue.

results-20251208-122718.zip

---

This is ready to be merged from my perspective. Any objections @pebrc, @kvalliyurnatt?

[kvalliyurnatt]

Follow-up

I've created #8945 to track the enhancement for improved error attribution in multi-SCP status reporting.

I ran scale testing with 100 SCPs targeting a single Elasticsearch cluster. Here are the key findings:

Test Steps:

1. Deploy 100 StackConfigPolicies targeting one ES cluster
2. Add a label change to the Elasticsearch CR to trigger reconciliation of all SCPs
3. Delete 50 SCPs
4. Delete remaining 50 SCPs

Results - Optimistic Concurrency Conflicts: The system handled 100 SCPs gracefully. Conflicts occurred as expected with optimistic concurrency control and were resolved through automatic retries:

• Step 1 (100 SCPs deployed): 13 conflicts on secrets
• Step 2 (ES label change, full SCP reconciliation): 0 conflicts
• Step 3 (50 SCPs deleted): 45 conflicts on secrets
• Step 4 (all SCPs deleted): 38 conflicts on secrets

All conflicts were transient "object has been modified" errors that resolved via retry. The annotation size limit (256KB) was not an issue.

results-20251208-122718.zip

This is ready to be merged from my perspective. Any objections @pebrc, @kvalliyurnatt?

Thanks for the detailed analysis. LGTM