[Infra UI] Stabilize Infra Custom Dashboards stateful API tests

[awahab07]

Summary

The Deployment Agnostic Infra Custom Dashboards API tests lately got many test failures in the scheduled stateful runs.

The most probably cause is a result of #262618, which introduced caching layer around uiSettings. As a result, Infra Custom Dashboards stateful API test suite which frequently updates and retrieves dashboard settings, faced server-side eventual consistency for settings in multi-node setups.

The suite was updating observability:enableInfrastructureAssetCustomDashboards and immediately asserting route behavior. In scheduled cloud runs, the follow-up request can observe a stale cached value on another Kibana node, which flips expected status codes.

Example failure:

This PR:

• stabilizes the suite by reducing advanced-setting flips.
• groups disabled and enabled scenarios so the suite only transitions into the enabled state once.
• waits for the enabled setting to propagate before running the enabled-path assertions (following ref).

[elasticmachine]

Pinging @elastic/obs-presentation-team (Team:obs-presentation)

[macroscopeapp]

Catch flakiness early (recommended)

Recommended before merge: run the flaky test runner against this PR to catch flakiness early.

The new waitForEnabledCustomDashboardsSetting retry/timing logic (retry.tryForTime with a 12 s propagation delay and 20 s timeout) and the restructured shared before/after hooks introduce non-determinism that a single CI pass may not reliably catch.

Trigger a run with the Flaky Test Runner UI or post this comment on the PR:

/flaky ftrConfig:x-pack/solutions/observability/test/api_integration_deployment_agnostic/configs/stateful/oblt.stateful.config.ts:30 ftrConfig:x-pack/solutions/observability/test/api_integration_deployment_agnostic/configs/serverless/oblt.serverless.config.ts:30

^{Share feedback in the #appex-qa channel.}

^{Posted via Macroscope — Flaky Test Runner nudge}

[macroscopeapp]

Approvability

Verdict: Needs human review

Test-only changes reorganizing API integration tests and adding retry logic for multi-node environment stability. However, the modified file is owned by @elastic/obs-presentation-team and the author is not a designated owner, so designated reviewers should verify the test reorganization.

^{You can customize Macroscope's approvability policy. Learn more.}

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 58d65ee

Metrics [docs]

✅ unchanged

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#11925

[✅] x-pack/solutions/observability/test/api_integration_deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 30/30 tests passed.

see run history

[jennypavlova]

Can we avoid having conditions in the tests?

[awahab07]

Yes makes sense. This shouldn't be the way to do it, but gives us a probe into the multi node environment failures.

I've created a follow up PR to remove this and hopefully after this we'll have a deterministic and robust way for such probe.

[jennypavlova]

Q: Any way this could be done here instead of leaving the comment. If not maybe we should link an issue so we don't forget about it

[awahab07]

Yes, makes sense.

I've created a follow up PR to remove this and created #265720.

[awahab07]

@jennypavlova thanks for the review. Does it make sense to let this PR merge, to test if it mitigates the recurring test failures. Once confirmed, we can move forward with the follow up PR and a robust solution tracked by #265720.