fix: race condition for pre-existing StackConfigPolicy

[pkoutsovasilis]

Summary

Reduces the time window where Elasticsearch clusters start with empty file-settings when a pre-existing
StackConfigPolicy should be applied, by coordinating reconciliation between ES and SCP controllers.

Problem

Prior to this PR:

1. User creates a StackConfigPolicy that targets ES clusters
2. User creates an Elasticsearch cluster matching the SCP resource selector
3. ES controller reconciles first and creates an empty file-settings secret
4. ES cluster starts and runs with empty file-settings
5. SCP controller eventually reconciles and updates the file-settings secret with policy configurations
6. ES cluster must reload/restart to apply the policy settings

This creates a non-negligible time window where the ES cluster operates without the intended policy configurations,
even though the policy existed before the cluster was created.

Solution

The Elasticsearch controller now coordinates with the StackConfigPolicy controller to ensure policy settings are
applied from the start when pre-existing policies target a new ES cluster.

New Reconciliation Flow

When enterprise features are disabled:

• ES controller creates an empty file-settings secret immediately (no change)

When enterprise features are enabled:

1. ES controller checks for targeting policies:

   • Lists all StackConfigPolicies in the cluster
   • Checks if any policy targets this ES cluster

2. If a pre-existing policy targets the cluster:

   • ES controller defers file-settings secret creation to the SCP controller
   • Returns re-queue to check again later
   • Result: SCP controller creates the secret with policy settings applied from the start

3. If no policy targets the cluster:

   • ES controller creates an empty file-settings secret as before

Testing

• Unit tests coverage for policy targeting scenarios
  • Verify requeue behaviour when policies target ES clusters
  • Confirm empty secret creation when no policies exist
• Manual testing with pre-existing StackConfigPolicies confirmed policy applied at cluster creation

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[barkbay]

LGTM

[barkbay]

buildkite test this -f p=gke,t=TestStackConfigPolicy*

[pebrc]

LGTM!