[kbn-es] Add --docker flag to yarn es snapshot

[patrykkopycinski]

Summary

Adds a --docker flag to yarn es snapshot that runs Elasticsearch in a Docker container with 1:1 equivalent behavior to the local snapshot flow, and wires Docker-based ES as the default mode for non-serverless Cypress tests (opt-out via CYPRESS_ES_FROM=snapshot).

What changed

Cypress tests: Docker ES as default for non-serverless

• Non-serverless Cypress tests now use esFrom: 'docker' by default, starting ES in a Docker container instead of downloading and extracting a local snapshot
• Set CYPRESS_ES_FROM=snapshot to revert to the previous behavior
• Serverless Cypress tests are unaffected — they continue using esFrom: 'serverless'

New esFrom: 'docker' mode for test infrastructure

• createTestEsCluster now supports esFrom: 'docker', starting ES in a Docker container instead of downloading and extracting a local snapshot
• runDockerSnapshotContainer accepts name (unique container naming for parallel safety) and background (non-blocking mode for test integration)
• Cluster class tracks Docker snapshot containers and handles cleanup (kill + rm) on stop
• New stopDockerSnapshotContainer() utility for explicit container teardown
• Transport port resolution unified with TEST_ES_TRANSPORT_PORT env var, matching the existing esTestConfig.getTransportPort() behavior (parses first port from range strings like '9300-9400')

New --docker mode for yarn es snapshot

• --docker — run ES in Docker instead of downloading locally
• --port — bind port (default 9200, Docker mode only)
• --kill — kill existing ES containers before starting (Docker mode only)
• All existing flags work: --license, --password, --ssl, -E, --skip-ready-check, --ready-timeout
• -E path.data=<relative-path> is automatically mapped to a Docker volume mount for data persistence
• -E values that reference host files (e.g. JWT JWKS keys) are automatically volume-mounted into the container with path rewriting
• --license=trial is mapped to xpack.license.self_generated.type=trial
• Same default esArgs as the local snapshot flow (action.destructive_requires_name, cluster.routing.allocation.disk.threshold_enabled=false, etc.)
• Waits for cluster readiness and sets up native realm passwords, matching local behavior

Usage example:

# Default for non-serverless (Docker ES)
node scripts/run_cypress parallel --config ...

# Opt-out to local snapshot
CYPRESS_ES_FROM=snapshot node scripts/run_cypress parallel --config ...

# yarn es snapshot with Docker
yarn es snapshot --docker -E path.data=../my-data --license=trial

Docker networking & container management

• ES binds to 0.0.0.0 inside the container for Fleet Server / Elastic Agent connectivity from other containers and VMs
• --add-host host.docker.internal:host-gateway added to ES containers for cross-cluster seed resolution
• cluster.remote.*.seeds esArgs automatically rewritten from localhost to host.docker.internal inside Docker
• Fleet Server uses host.docker.internal for ES connectivity when ES runs in Docker
• CCS transport port (transportPort) properly mapped to container port 9300
• Stale containers are force-removed before starting a new one (handles pRetry retries cleanly)
• Only verifyDockerInstalled() + maybeCreateDockerNetwork() are called for Docker snapshot setup — avoids interfering with serverless containers (no detectRunningNodes / cleanUpDanglingContainers)

Expected boot time gains (when using Docker)

Running ES via Docker eliminates the snapshot download + extraction overhead. With a pre-pulled image the gains are significant:

Phase	Snapshot (local)	Docker	Savings
Download archive	~30-60s (cold) / 0s (cached)	0s (image cached)	up to ~60s
Extract / install	~10-20s	0s	~10-20s
Container startup	N/A	~2-5s	—
Cluster ready check	~10-15s	~10-15s	~0s
Native realm setup	~2-3s	~2-3s	~0s
Total (cold)	~55-100s	~15-25s	~40-75s
Total (warm)	~25-40s	~15-25s	~10-20s

Key observations:

• Cold start (first run / CI with no cache): Docker avoids the full snapshot download (~200MB) and extraction. The Docker image is pulled once and cached across runs.
• Warm start (snapshot already cached locally): Docker still saves ~10-20s by skipping archive extraction and JVM bootstrap overhead.
• CI impact: Requires Docker-in-Docker support on CI agents. When available, the Docker image can be pre-cached in the base image or pulled in parallel with other setup steps.
• Per-spec overhead: Since the Cypress parallel runner starts/stops ES for each spec file, savings compound — e.g. 10 spec files × 15s saved = ~2.5 minutes total.

native_realm.js → native_realm.ts migration

• Full TypeScript conversion with typed constructor (NativeRealmOptions), retry options (RetryOpts), and all method signatures
• Removed legacy body wrapper on ES client API calls (not needed with newer @elastic/elasticsearch client)
• Converted from mixed CJS/ESM to pure ESM, removing the @ts-expect-error suppression in the barrel export

Cleanup

• Removed stale arm64 -XX:UseSVE=0 JVM workaround (elastic/elasticsearch#118583) — no longer needed at ES 9.x
• Extracted duplicated Java opts logic into the inline arrays directly

Files changed

File	Change
kbn-es/src/cli_commands/snapshot.ts	Added --docker, --port, --kill flags
kbn-es/src/cluster.ts	Added runDockerSnapshot() method, Docker container tracking and cleanup
kbn-es/src/cluster_exec_options.ts	Added Docker-related exec option types
kbn-es/src/utils/docker.ts	Added runDockerSnapshotContainer() with name/background/transportPort options, stopDockerSnapshotContainer(), host file volume mounting, host.docker.internal support, removed arm64 workaround
kbn-es/src/utils/docker_uiam.ts	Updated for Docker snapshot compatibility
kbn-es/src/utils/index.ts	Removed @ts-expect-error for native_realm
kbn-es/src/utils/native_realm.js → .ts	TypeScript migration
kbn-es/src/utils/native_realm.test.js → .ts	TypeScript migration
kbn-test/src/es/test_es_cluster.ts	Added esFrom: 'docker' branch, unified transport port resolution with TEST_ES_TRANSPORT_PORT
security_solution/scripts/run_cypress/parallel.ts	Default to Docker ES for non-serverless tests, CYPRESS_ES_FROM env var for opt-out

Risk

Low — additive feature behind new flags/env vars. Serverless tests are untouched. Non-serverless Cypress tests default to Docker but can opt out via CYPRESS_ES_FROM=snapshot. The native_realm migration preserves identical runtime behavior.

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[patrykkopycinski]

/ci

[gergoabraham]

looks amazing, thanks for improving on the test env! 🙇

@elastic/security-defend-workflows related changes look good 👍

[ashokaditya]

Thanks for adding changes from the previous PR review! Tested this out and works as expected.

[dmlemeshko]

this is not following the existing logic, maybe we can unify it:

kibana/src/platform/packages/shared/kbn-test/src/es/es_test_config.ts

Lines 35 to 37 in 773bb6a

	getTransportPort() {
	return process.env.TEST_ES_TRANSPORT_PORT || '9300-9400';
	}

[dmlemeshko]

LGTM, left a nit about TEST_ES_TRANSPORT_PORT variable.

Please double check with Ops Team about potential impact.

[patrykkopycinski]

/ci

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 76e1af3

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/es	40	41	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/es	7	8	+1

Unknown metric groups

API count

id	before	after	diff
@kbn/es	55	57	+2

History

• 💔 Build #399734 failed 124a951
• 💚 Build #398939 succeeded 5efac7e
• 💔 Build #398923 failed f4b0118
• 💔 Build #398919 failed 39730fd
• 💔 Build #398909 failed 4d33ac0
• 💔 Build #398906 failed 5d484e2

cc @patrykkopycinski

[delanni]

Unfortunately this introduces even more modes to run ES in kibana, which will surely be more confusion, but I understand it's beneficial for a multi-repo usecase 👍