Skip to content

[scout] update default server configs to use dockerized package registry#255484

Draft
dmlemeshko wants to merge 4 commits intoelastic:mainfrom
dmlemeshko:scout/erp-container-integration
Draft

[scout] update default server configs to use dockerized package registry#255484
dmlemeshko wants to merge 4 commits intoelastic:mainfrom
dmlemeshko:scout/erp-container-integration

Conversation

@dmlemeshko
Copy link
Copy Markdown
Contributor

Summary

Migrates Scout default configs to use the centralized Fleet dockerized package registry from @kbn/test (introduced in #250040), adds Docker container lifecycle management to Scout's server startup flow, and wires --xpack.fleet.registryUrl to Kibana when the registry is enabled.

Changes

  • Use centralized packageRegistryDocker from @kbn/test in both stateful and serverless base configs, replacing duplicated inline configuration. Scout overrides only waitForLogLineTimeoutMs (6 min vs 4 min default) for CI stability.
  • Add --xpack.fleet.registryUrl to kbnTestServer.serverArgs when FLEET_PACKAGE_REGISTRY_PORT is set, matching the FTR pattern. Custom Scout configs can override this via the standard spread pattern.
  • New startDockerServers module (run_docker_servers.ts) -- a lightweight, standalone async function that pulls images, starts containers, waits for readiness, and returns a shutdown callback. No coupling to FTR's Lifecycle/Rx machinery.
  • Integrate Docker startup into run_tests.ts and start_servers.ts (after ES, before Kibana) so the registry is available at Kibana boot.
  • Remove unused package_registry_config.yml from Scout config directories (now bundled with @kbn/test).

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 2, 2026

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!
  • Click to trigger kibana-deploy-cloud-from-pr for this PR!
  • Click to trigger kibana-entity-store-performance-from-pr for this PR!
  • Click to trigger kibana-storybooks-from-pr for this PR!

@dmlemeshko dmlemeshko changed the title [scout] update default condigs to use fleet lite container [scout] update default server configs to use dockerized package registry Mar 2, 2026
@dmlemeshko dmlemeshko force-pushed the scout/erp-container-integration branch from fd96162 to 080ad16 Compare April 21, 2026 15:31
@macroscopeapp
Copy link
Copy Markdown
Contributor

macroscopeapp Bot commented Apr 21, 2026

Scout Test Review

No issues found ✅

All changed files under **/kbn-scout*/** are internal framework implementation and unit tests (run_docker_servers.ts, run_docker_servers.test.ts, base_config.test.ts, config files). No Scout test specs, page objects, API services, or fixtures were modified.

Share feedback in the #appex-qa channel.

Posted via Macroscope — Scout Test Review

@@ -46,6 +47,8 @@ export async function startServers(log: ToolingLog, options: StartServerOptions)

await ensureDefaultSpaceNPRE(config, log);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High servers/start_servers.ts:48

If any code between startDockerServers (line 50) and procs.waitForAllToStop (line 80) throws an exception, shutdownDockerServers() at line 81 never runs and Docker containers are left running. Move shutdownDockerServers into a finally block or wrap the intervening code in try/finally so cleanup executes regardless of errors.

Also found in 1 other location(s)

src/platform/packages/shared/kbn-scout/src/servers/run_docker_servers.ts:170

If pullImage, runContainer, or waitForReady throws an exception for any server after previous containers have already been started, those containers will be orphaned. The runningContainers array accumulates started containers, but if an error occurs mid-loop, the function throws before returning the shutdown function, leaving earlier containers running with no way to clean them up. For example, if servers A and B are enabled, A starts successfully and is pushed to runningContainers, but B fails during runContainer (e.g., port conflict as handled on line 61-65), container A is never stopped.

🤖 Copy this AI Prompt to have your agent fix this:
In file src/platform/packages/shared/kbn-scout/src/servers/start_servers.ts around line 48:

If any code between `startDockerServers` (line 50) and `procs.waitForAllToStop` (line 80) throws an exception, `shutdownDockerServers()` at line 81 never runs and Docker containers are left running. Move `shutdownDockerServers` into a `finally` block or wrap the intervening code in `try`/`finally` so cleanup executes regardless of errors.

Evidence trail:
src/platform/packages/shared/kbn-scout/src/servers/start_servers.ts lines 50-82 at REVIEWED_COMMIT. The `shutdownDockerServers` function is obtained at line 50, various operations that could throw are executed at lines 52-76 (runKibanaServer, silence, preCreateSecurityIndexesViaSamlAuth), and `shutdownDockerServers()` is only called at line 81. There is no try/finally block ensuring cleanup runs on exception.

Also found in 1 other location(s):
- src/platform/packages/shared/kbn-scout/src/servers/run_docker_servers.ts:170 -- If `pullImage`, `runContainer`, or `waitForReady` throws an exception for any server after previous containers have already been started, those containers will be orphaned. The `runningContainers` array accumulates started containers, but if an error occurs mid-loop, the function throws before returning the shutdown function, leaving earlier containers running with no way to clean them up. For example, if servers A and B are enabled, A starts successfully and is pushed to `runningContainers`, but B fails during `runContainer` (e.g., port conflict as handled on line 61-65), container A is never stopped.

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 22, 2026

⏳ Build in-progress, with failures

Failed CI Steps

Test Failures

  • [job] [logs] affected Scout: [ platform / gen_ai_settings ] plugin / local-serverless-observability_complete - GenAI Settings - Documentation Section - should show Documentation section in Agent mode and hide it in Classic mode
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-serverless-observability_complete - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-serverless-observability_complete - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-serverless-security_complete - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-serverless-security_complete - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ platform / fleet ] plugin / local-stateful-classic - Browse integration - it allow to sort integrations
  • [job] [logs] affected Scout: [ platform / fleet ] plugin / local-stateful-classic - Browse integration - it allow to sort integrations
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-stateful-classic - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-stateful-classic - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-stateful-classic - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-stateful-classic - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] Jest Tests #8 / SelectedFilters should render properly

History

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Apr 30, 2026

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #4 / getThrottleStrategyLabel returns the aggregate every-time label when mode is all
  • [job] [logs] Jest Tests #4 / getThrottleStrategyLabel returns the aggregate every-time label when mode is all
  • [job] [logs] Jest Tests #4 / getThrottleStrategyLabel returns the per-episode every-time label when mode is per_episode
  • [job] [logs] Jest Tests #4 / getThrottleStrategyLabel returns the per-episode every-time label when mode is per_episode
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-serverless-observability_complete - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-serverless-observability_complete - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-serverless-observability_complete - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-serverless-security_complete - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-serverless-security_complete - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ platform / fleet ] plugin / local-stateful-classic - Browse integration - it allow to sort integrations
  • [job] [logs] affected Scout: [ platform / fleet ] plugin / local-stateful-classic - Browse integration - it allow to sort integrations
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-stateful-classic - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / infra ] plugin / local-stateful-classic - Infrastructure Inventory - K8s Integration Links Navigation - K8s dashboards link navigates to dashboards with tag filter
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Onboarding UI Validation - navigates correctly within Kubernetes Host flow
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-stateful-classic - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ platform / osquery ] plugin / local-stateful-classic - Osquery packs - policy ID validation - strips duplicate policy IDs and returns 200
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - command includes wired streams config when Wired Streams mode is selected
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - Elastic Agent Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options
  • [job] [logs] affected Scout: [ observability / observability_onboarding ] plugin / local-stateful-classic - Wired Streams - OTel Kubernetes Flow - shows ingestion selector with correct options

Metrics [docs]

✅ unchanged

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants