[Osquery] Cypress - Add SHA512 integrity validation for cached agent by tomsonpl · Pull Request #258842 · elastic/kibana

tomsonpl · 2026-03-20T13:25:36Z

Summary

Adds SHA512 integrity validation for cached Elastic Agent downloads used by Defend Workflows Cypress tests. Corrupt or truncated tarballs are now automatically detected and re-downloaded, preventing persistent CI failures.

Problem

The Defend Workflows Cypress tests (cy.task('createEndpointHost')) provision Vagrant VMs with Elastic Agent. The agent tarball is pre-downloaded during CI image build and cached at ~/.kibanaSecuritySolutionCliTools/agent_download_storage/.

Example
https://buildkite.com/elastic/kibana-on-merge/builds/91361#019d0abf-4e4e-43e2-a693-44b1023278b4/L2306-L2428

There was zero integrity validation anywhere in this chain. If a download was truncated (network hiccup, CDN issue), the partial file was cached and reused for up to 2 days, causing every test run on that CI agent to fail with:

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

The Elastic artifacts API already exposes sha_url alongside url in search responses, but it was completely ignored.

Changes

SHA512 validation after download (agent_downloads_service.ts): fetch expected hash from artifacts API sha_url, compute SHA512 of downloaded file, compare. On mismatch, retry (up to 3 attempts). On sha_url fetch failure, proceed without validation (best-effort — don't block CI if hash infra is down).
Cache integrity validation on reuse (agent_downloads_service.ts): cached files now require a .sha512 sidecar file. On cache hit, re-compute hash and compare with sidecar. If mismatch or missing sidecar → delete corrupt file and re-download. This is the key self-healing behavior.
SHA URL propagation (fleet_services.ts): getAgentDownloadUrl() now returns shaUrl from the artifacts API response. All callers updated to pass it through: agent_downloader CLI (packer cache), create_and_enroll_endpoint_host_ci.ts (Cypress), enrollHostVmWithFleet().
Vagrantfile defense-in-depth: added gzip -t check before tar -zxf extraction. Catches any corruption that slips through (e.g., SCP transfer issue). Fails with "Agent tarball integrity check failed" instead of cryptic tar errors.
Sidecar cleanup: .sha512 files are cleaned up alongside their parent tarballs during the existing cache TTL cleanup.

How it works

Before (no validation):
  cached file exists? ──yes──► use as-is ──► tar -zxf ──► 💥 Unexpected EOF

After (self-healing):
  cached file exists? ──yes──► has .sha512 sidecar? ──no──► delete, re-download
                                       │
                                      yes
                                       │
                                       ▼
                               compute hash, compare
                                       │
                              match? ──yes──► use it ✓
                                       │
                                      no
                                       │
                                       ▼
                               delete, re-download ──► validate ──► cache ✓

Test plan

agent_downloads_service.test.ts — 14 tests covering:
- Fresh download without cache
- Cache hit with valid sidecar hash
- Cache hit with invalid hash → re-download
- Cache hit with missing sidecar → re-download
- Hash validation after fresh download with shaUrl
- All retry attempts fail hash validation → error
- sha_url fetch fails → proceed without validation
- Sidecar file created on successful download
- Sidecar file deleted during cleanup
- isAgentDownloadFromDiskAvailable with/without sidecar
- fetchExpectedHash parsing and error handling
agent_downloader.test.ts — 8 tests verify shaUrl passthrough in all scenarios
All 22 tests pass

…c Agent downloads

tomsonpl · 2026-03-20T13:27:44Z

/ci

elasticmachine · 2026-03-20T13:30:04Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

szwarckonrad

Two things I noticed:

computeFileHash will hang at runtime

await finished(stream.pipe(hash)) never resolves because Hash is a Transform stream and nobody reads from its readable side, so finished() waits forever. Tests pass because createHash is fully mocked. I verified locally on Node 22, it hangs.

Quick fix:

export const computeFileHash = async (filePath: string): Promise<string> => {
  const hash = createHash('sha512');
  const stream = fs.createReadStream(filePath);
  for await (const chunk of stream) {
    hash.update(chunk);
  }
  return hash.digest('hex');
};

Cache breaks when the SHA URL is down

If fetchExpectedHash fails, expectedHash stays undefined, so no sidecar file gets written. Next run sees the file but no sidecar, treats it as unavailable, and re-downloads. This loops forever while the hash endpoint is unreachable, effectively disabling the cache.

One option: compute the local hash after download and store it as the sidecar regardless, so at least disk-level corruption is caught on reuse even without remote validation.

…ix-corrupted-agent-file

elasticmachine · 2026-03-23T09:53:08Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: 8577606

Failed CI Steps

History

cc @tomsonpl

tomsonpl · 2026-03-23T12:05:16Z

Thanks @szwarckonrad, my implementation wasnt full without your suggestion. Big thank you! Fixed now :)

kibanamachine · 2026-03-23T17:43:35Z

Starting backport for target branches: 8.19, 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/23451616679

kibanamachine · 2026-03-23T17:51:58Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.2	Backport failed because of merge conflicts
❌	9.3	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 258842