Skip to content

[CI] Harden Defend Workflows VM provisioning#254354

Merged
patrykkopycinski merged 3 commits intoelastic:mainfrom
patrykkopycinski:harden-defend-workflows-ci
Feb 23, 2026
Merged

[CI] Harden Defend Workflows VM provisioning#254354
patrykkopycinski merged 3 commits intoelastic:mainfrom
patrykkopycinski:harden-defend-workflows-ci

Conversation

@patrykkopycinski
Copy link
Copy Markdown
Contributor

@patrykkopycinski patrykkopycinski commented Feb 22, 2026 •
edited by kibanamachine
Loading

Summary

Fixes Defend Workflows Cypress CI failures caused by VirtualBox kernel module not being loaded on CI agents. The CI image ships with VirtualBox 7.0.x, but recent GCP kernel updates (6.17.0-1008-gcp) break module compilation for that version.

Changes

New: ensure_virtualbox.sh — shared pre-flight script sourced by all Defend Workflows CI scripts. It:

  1. Checks if VBoxManage --version reports a healthy state (not just exit code — parses output for "kernel module is not loaded" warning)
  2. Attempts to load existing kernel modules (modprobe vboxdrv, /sbin/vboxconfig)
  3. If module build fails (kernel too new for VirtualBox 7.0.x), upgrades to VirtualBox 7.1 from Oracle's repo which supports newer kernels
  4. Exports VAGRANT_DEFAULT_PROVIDER=virtualbox to skip Vagrant's provider auto-discovery
  5. Prints full diagnostics on failure (packages, kernel version, DKMS status)

Updated: vm_services.ts

  • VAGRANT_DEFAULT_PROVIDER=virtualbox set in vagrant env on CI (local dev keeps VMware auto-discovery)
  • Pre-flight disk space check

Updated: Vagrantfile

  • Installs unzip via apt-get for provisioning

Files changed

  • .buildkite/scripts/steps/functional/ensure_virtualbox.sh (new)
  • .buildkite/scripts/steps/functional/defend_workflows*.sh (4 files — source ensure_virtualbox.sh)
  • x-pack/.../endpoint/common/vm_services.ts
  • x-pack/.../endpoint/common/vagrant/Vagrantfile

@patrykkopycinski
[CI] Harden Defend Workflows VM provisioning and bump disk to 120GB
aaaa2d1 
- Bump diskSizeGb from 105 to 120 across all DW pipeline configs
- Add pre-flight disk space check before vagrant up (fail fast <10GB)
- Add vagrant up retry logic (2 attempts) with cleanup between retries
- Replace apt-get install unzip with python3 zipfile wrapper (no network)
- Harden Vagrantfile: mkdir -p, capture stdout/stderr for diagnostics
@patrykkopycinski patrykkopycinski added release_note:skip Skip the PR/issue when compiling release notes ci:all-cypress-suites backport:all-open Backport to all branches that could still receive a release labels Feb 22, 2026
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

1 similar comment
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

@patrykkopycinski patrykkopycinski force-pushed the harden-defend-workflows-ci branch from dfe64ce to bc41042 Compare February 22, 2026 22:33
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

@patrykkopycinski patrykkopycinski force-pushed the harden-defend-workflows-ci branch from bc41042 to 2fbd209 Compare February 22, 2026 22:36
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

patrykkopycinski added a commit to patrykkopycinski/kibana that referenced this pull request Feb 22, 2026
@patrykkopycinski
Align DW hardening with PR elastic#254354 latest state
39730fd 
- Add ensure_virtualbox.sh: checks, recovers, or installs VirtualBox
- Source ensure_virtualbox.sh in all 4 DW shell scripts
- Set VAGRANT_DEFAULT_PROVIDER=virtualbox on CI
- Add ensureVirtualBoxProvider() in vm_services.ts as defense-in-depth
- Pre-flight disk space check before vagrant up
- Vagrant up retry logic (2 attempts) with cleanup between retries
- Replace apt-get install unzip with python3 zipfile wrapper (no network)
- Proper TypeScript error type casts
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

@patrykkopycinski patrykkopycinski force-pushed the harden-defend-workflows-ci branch from eced2de to 00a9766 Compare February 22, 2026 23:44
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 22, 2026

/ci

@patrykkopycinski patrykkopycinski force-pushed the harden-defend-workflows-ci branch from 00a9766 to a338326 Compare February 23, 2026 00:19
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 23, 2026

/ci

@patrykkopycinski patrykkopycinski changed the title [CI] Harden Defend Workflows VM provisioning and bump disk to 120GB [CI] Harden Defend Workflows VM provisioning with VirtualBox auto-recovery Feb 23, 2026
@patrykkopycinski patrykkopycinski changed the title [CI] Harden Defend Workflows VM provisioning with VirtualBox auto-recovery [CI] Harden Defend Workflows VM provisioning Feb 23, 2026
@patrykkopycinski patrykkopycinski marked this pull request as ready for review February 23, 2026 01:08
@patrykkopycinski patrykkopycinski requested review from a team as code owners February 23, 2026 01:08
@patrykkopycinski
[CI] Ensure VirtualBox provider on CI before Defend Workflows tests
83a1856 
- Add ensure_virtualbox.sh: checks, recovers, or installs VirtualBox
- Default VAGRANT_DEFAULT_PROVIDER=virtualbox on CI to skip auto-discovery
- Add ensureVirtualBoxProvider() in vm_services.ts as defense-in-depth
- Keep local runs unchanged (VMware auto-discovery preserved)
@patrykkopycinski
Copy link
Copy Markdown
Contributor Author

patrykkopycinski commented Feb 23, 2026

/ci

@patrykkopycinski patrykkopycinski force-pushed the harden-defend-workflows-ci branch from a338326 to 83a1856 Compare February 23, 2026 01:15
patrykkopycinski added a commit to patrykkopycinski/kibana that referenced this pull request Feb 23, 2026
@patrykkopycinski
Align DW hardening with PR elastic#254354 latest state
5efac7e 
- Improved ensure_virtualbox.sh: kernel module check, VBox 7.1 upgrade
  path, DKMS diagnostics
- Vagrantfile: use simple apt-get install unzip (matching PR)
- vm_services.ts: refined ensureVirtualBoxProvider with proper kernel
  module detection and VBox 7.1 upgrade fallback
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Feb 23, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 720 719 -1

Total ESLint disabled count

id before after diff
securitySolution 825 824 -1

History

mistic
mistic reviewed Feb 23, 2026
View reviewed changes
Comment thread .buildkite/scripts/steps/functional/ensure_virtualbox.sh
}

_upgrade_to_vbox71() {
echo "Upgrading to VirtualBox 7.1 (supports newer kernels)..."
Copy link
Copy Markdown
Contributor

@mistic mistic Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it better to make sure CI images ship with Virtual Box 7.1? Should we add a TODO / open a ticket for it?

Copy link
Copy Markdown
Contributor Author

@patrykkopycinski patrykkopycinski Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely having it on CI images is a better longterm solution, but I think it would make sense to keep it here as well, so we have a better logs if it starts failing again in the future

@patrykkopycinski patrykkopycinski enabled auto-merge (squash) February 23, 2026 07:42
@patrykkopycinski patrykkopycinski self-assigned this Feb 23, 2026
ashokaditya
ashokaditya approved these changes Feb 23, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@ashokaditya ashokaditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!! I've a few suggestions but otherwise LGTM. 🚢 it!

Comment thread x-pack/solutions/security/plugins/security_solution/scripts/endpoint/common/vm_services.ts
return;
}

const loadModules = async (): Promise<boolean> => {
Copy link
Copy Markdown
Member

@ashokaditya ashokaditya Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider renaming this to isVirtualBoxLoaded or areModulesLoaded

Comment thread x-pack/solutions/security/plugins/security_solution/scripts/endpoint/common/vm_services.ts
log?: ToolingLog;
}

const ensureVirtualBoxProvider = async (log: ToolingLog): Promise<void> => {
Copy link
Copy Markdown
Member

@ashokaditya ashokaditya Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving CreateVagrantVmOptions type close to thecreateVagrantHostVmClient function below.

Comment thread x-pack/solutions/security/plugins/security_solution/scripts/endpoint/common/vm_services.ts
if (await upgradeToVbox71()) {
return;
}

Copy link
Copy Markdown
Member

@ashokaditya ashokaditya Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we log here that upgrade was successful?

@patrykkopycinski patrykkopycinski merged commit 11333fb into elastic:main Feb 23, 2026
16 checks passed
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Feb 23, 2026

Starting backport for target branches: 8.19, 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/22300406621

patrykkopycinski added a commit to patrykkopycinski/kibana that referenced this pull request Feb 23, 2026
@patrykkopycinski
Address PR elastic#254354 review feedback
945347c 
- Rename loadModules to tryLoadModules for clarity
- Move CreateVagrantVmOptions interface next to createVagrantVm
- Log success message after VirtualBox 7.1 upgrade
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Feb 23, 2026
@patrykkopycinski @kibanamachine
[CI] Harden Defend Workflows VM provisioning (elastic#254354)
e3ddd1e 
## Summary

Fixes Defend Workflows Cypress CI failures caused by VirtualBox kernel
module not being loaded on CI agents. The CI image ships with VirtualBox
7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`) break module
compilation for that version.

### Changes

**New: `ensure_virtualbox.sh`** — shared pre-flight script sourced by
all Defend Workflows CI scripts. It:
1. Checks if `VBoxManage --version` reports a healthy state (not just
exit code — parses output for "kernel module is not loaded" warning)
2. Attempts to load existing kernel modules (`modprobe vboxdrv`,
`/sbin/vboxconfig`)
3. If module build fails (kernel too new for VirtualBox 7.0.x), upgrades
to **VirtualBox 7.1** from Oracle's repo which supports newer kernels
4. Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip Vagrant's
provider auto-discovery
5. Prints full diagnostics on failure (packages, kernel version, DKMS
status)

**Updated: `vm_services.ts`**
- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in vagrant env on CI (local
dev keeps VMware auto-discovery)
- Pre-flight disk space check

**Updated: `Vagrantfile`**
- Installs `unzip` via `apt-get` for provisioning

### Files changed
- `.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)
- `.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files —
source ensure_virtualbox.sh)
- `x-pack/.../endpoint/common/vm_services.ts`
- `x-pack/.../endpoint/common/vagrant/Vagrantfile`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 11333fb)
@kibanamachine kibanamachine mentioned this pull request Feb 23, 2026
Merged
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Feb 23, 2026
@patrykkopycinski @kibanamachine
[CI] Harden Defend Workflows VM provisioning (elastic#254354)
3784aab 
## Summary

Fixes Defend Workflows Cypress CI failures caused by VirtualBox kernel
module not being loaded on CI agents. The CI image ships with VirtualBox
7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`) break module
compilation for that version.

### Changes

**New: `ensure_virtualbox.sh`** — shared pre-flight script sourced by
all Defend Workflows CI scripts. It:
1. Checks if `VBoxManage --version` reports a healthy state (not just
exit code — parses output for "kernel module is not loaded" warning)
2. Attempts to load existing kernel modules (`modprobe vboxdrv`,
`/sbin/vboxconfig`)
3. If module build fails (kernel too new for VirtualBox 7.0.x), upgrades
to **VirtualBox 7.1** from Oracle's repo which supports newer kernels
4. Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip Vagrant's
provider auto-discovery
5. Prints full diagnostics on failure (packages, kernel version, DKMS
status)

**Updated: `vm_services.ts`**
- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in vagrant env on CI (local
dev keeps VMware auto-discovery)
- Pre-flight disk space check

**Updated: `Vagrantfile`**
- Installs `unzip` via `apt-get` for provisioning

### Files changed
- `.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)
- `.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files —
source ensure_virtualbox.sh)
- `x-pack/.../endpoint/common/vm_services.ts`
- `x-pack/.../endpoint/common/vagrant/Vagrantfile`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 11333fb)
@kibanamachine kibanamachine mentioned this pull request Feb 23, 2026
Merged
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Feb 23, 2026
@patrykkopycinski @kibanamachine
[CI] Harden Defend Workflows VM provisioning (elastic#254354)
6dc0788 
## Summary

Fixes Defend Workflows Cypress CI failures caused by VirtualBox kernel
module not being loaded on CI agents. The CI image ships with VirtualBox
7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`) break module
compilation for that version.

### Changes

**New: `ensure_virtualbox.sh`** — shared pre-flight script sourced by
all Defend Workflows CI scripts. It:
1. Checks if `VBoxManage --version` reports a healthy state (not just
exit code — parses output for "kernel module is not loaded" warning)
2. Attempts to load existing kernel modules (`modprobe vboxdrv`,
`/sbin/vboxconfig`)
3. If module build fails (kernel too new for VirtualBox 7.0.x), upgrades
to **VirtualBox 7.1** from Oracle's repo which supports newer kernels
4. Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip Vagrant's
provider auto-discovery
5. Prints full diagnostics on failure (packages, kernel version, DKMS
status)

**Updated: `vm_services.ts`**
- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in vagrant env on CI (local
dev keeps VMware auto-discovery)
- Pre-flight disk space check

**Updated: `Vagrantfile`**
- Installs `unzip` via `apt-get` for provisioning

### Files changed
- `.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)
- `.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files —
source ensure_virtualbox.sh)
- `x-pack/.../endpoint/common/vm_services.ts`
- `x-pack/.../endpoint/common/vagrant/Vagrantfile`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 11333fb)
@kibanamachine kibanamachine mentioned this pull request Feb 23, 2026
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Feb 23, 2026

💚 All backports created successfully

Status Branch Result
✅ 8.19
✅ 9.2
✅ 9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Feb 23, 2026
@kibanamachine @patrykkopycinski
[9.3] [CI] Harden Defend Workflows VM provisioning (#254354) (#254384)
6c00d51 
# Backport

This will backport the following commits from `main` to `9.3`:
- [[CI] Harden Defend Workflows VM provisioning
(#254354)](#254354)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Patryk
Kopyciński","email":"contact@patrykkopycinski.com"},"sourceCommit":{"committedDate":"2026-02-23T09:36:04Z","message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:all-cypress-suites","backport:all-open","v9.4.0"],"title":"[CI]
Harden Defend Workflows VM
provisioning","number":254354,"url":"https://github.com/elastic/kibana/pull/254354","mergeCommit":{"message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/254354","number":254354,"mergeCommit":{"message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee"}}]}]
BACKPORT-->

Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
kibanamachine added a commit that referenced this pull request Feb 23, 2026
@kibanamachine @patrykkopycinski
[9.2] [CI] Harden Defend Workflows VM provisioning (#254354) (#254383)
3154a5b 
# Backport

This will backport the following commits from `main` to `9.2`:
- [[CI] Harden Defend Workflows VM provisioning
(#254354)](#254354)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Patryk
Kopyciński","email":"contact@patrykkopycinski.com"},"sourceCommit":{"committedDate":"2026-02-23T09:36:04Z","message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:all-cypress-suites","backport:all-open","v9.4.0"],"title":"[CI]
Harden Defend Workflows VM
provisioning","number":254354,"url":"https://github.com/elastic/kibana/pull/254354","mergeCommit":{"message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/254354","number":254354,"mergeCommit":{"message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee"}}]}]
BACKPORT-->

Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
kibanamachine added a commit that referenced this pull request Feb 23, 2026
@kibanamachine @patrykkopycinski
[8.19] [CI] Harden Defend Workflows VM provisioning (#254354) (#254382)
29656f7 
# Backport

This will backport the following commits from `main` to `8.19`:
- [[CI] Harden Defend Workflows VM provisioning
(#254354)](#254354)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Patryk
Kopyciński","email":"contact@patrykkopycinski.com"},"sourceCommit":{"committedDate":"2026-02-23T09:36:04Z","message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:all-cypress-suites","backport:all-open","v9.4.0"],"title":"[CI]
Harden Defend Workflows VM
provisioning","number":254354,"url":"https://github.com/elastic/kibana/pull/254354","mergeCommit":{"message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/254354","number":254354,"mergeCommit":{"message":"[CI]
Harden Defend Workflows VM provisioning (#254354)\n\n## Summary\n\nFixes
Defend Workflows Cypress CI failures caused by VirtualBox kernel\nmodule
not being loaded on CI agents. The CI image ships with
VirtualBox\n7.0.x, but recent GCP kernel updates (`6.17.0-1008-gcp`)
break module\ncompilation for that version.\n\n### Changes\n\n**New:
`ensure_virtualbox.sh`** — shared pre-flight script sourced by\nall
Defend Workflows CI scripts. It:\n1. Checks if `VBoxManage --version`
reports a healthy state (not just\nexit code — parses output for
\"kernel module is not loaded\" warning)\n2. Attempts to load existing
kernel modules (`modprobe vboxdrv`,\n`/sbin/vboxconfig`)\n3. If module
build fails (kernel too new for VirtualBox 7.0.x), upgrades\nto
**VirtualBox 7.1** from Oracle's repo which supports newer kernels\n4.
Exports `VAGRANT_DEFAULT_PROVIDER=virtualbox` to skip
Vagrant's\nprovider auto-discovery\n5. Prints full diagnostics on
failure (packages, kernel version, DKMS\nstatus)\n\n**Updated:
`vm_services.ts`**\n- `VAGRANT_DEFAULT_PROVIDER=virtualbox` set in
vagrant env on CI (local\ndev keeps VMware auto-discovery)\n- Pre-flight
disk space check\n\n**Updated: `Vagrantfile`**\n- Installs `unzip` via
`apt-get` for provisioning\n\n### Files changed\n-
`.buildkite/scripts/steps/functional/ensure_virtualbox.sh` (new)\n-
`.buildkite/scripts/steps/functional/defend_workflows*.sh` (4 files
—\nsource ensure_virtualbox.sh)\n-
`x-pack/.../endpoint/common/vm_services.ts`\n-
`x-pack/.../endpoint/common/vagrant/Vagrantfile`\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"11333fb52d6523a888326c1f2987e0f6ced060ee"}}]}]
BACKPORT-->

Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
patrykkopycinski added a commit to patrykkopycinski/kibana that referenced this pull request Feb 25, 2026
@patrykkopycinski
Cherry-pick: [CI] Harden Defend Workflows VM provisioning (elastic#25…
293dfa8 
…4354)

Cherry-pick from patrykkopycinski/kibana#254354 (83a1856):
- Add ensure_virtualbox.sh for VirtualBox kernel module recovery
- Source ensure_virtualbox.sh in all 4 DW shell scripts
- Set VAGRANT_DEFAULT_PROVIDER=virtualbox on CI
- Add ensureVirtualBoxProvider() in vm_services.ts
- Vagrant up retry logic with cleanup between attempts
- Combine apt-get update+install in Vagrantfile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashokaditya ashokaditya ashokaditya approved these changes

@mistic mistic mistic approved these changes

@joeypoon joeypoon Awaiting requested review from joeypoon joeypoon is a code owner automatically assigned from elastic/security-defend-workflows

@szwarckonrad szwarckonrad Awaiting requested review from szwarckonrad szwarckonrad is a code owner automatically assigned from elastic/security-defend-workflows

Assignees

@patrykkopycinski patrykkopycinski

Labels

backport:all-open Backport to all branches that could still receive a release ci:all-cypress-suites release_note:skip Skip the PR/issue when compiling release notes v8.19.12 v9.2.6 v9.3.1 v9.4.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@patrykkopycinski @elasticmachine @kibanamachine @ashokaditya @mistic