[CI] Filter empty jest tests before grouping by delanni · Pull Request #242440 · elastic/kibana

delanni · 2025-11-10T15:35:20Z

Summary

It seems the jest unit test grouping through ci-stats is broken. It's probably because we no longer run and report empty test configs (after #236549 we skip empty configs), thus CI stats will no longer be able to have an accurate guess as to how long they will take. Calculating with the default guessed config runtimes results in weird buckets. We went from ~23 jest unit test groups to ~53 (again, probably because the empty configs, that would run in a few seconds are counted as if they take a few minutes + overhead).

This PR updates the pick_test_group_run_order logic, such that we no longer ask grouping for the configs that don't contain any test files to run. The result reset the jest unit parallel group count back to 23.

For this, I'm using jest's internal utilities to search for test files, and had to bootstrap the step, so that the resolution doesn't break.

…because the step wasn't bootstrapped

elasticmachine · 2025-11-12T15:30:26Z

Pinging @elastic/kibana-operations (Team:Operations)

…, fix bad jest config

delanni · 2025-11-13T16:29:39Z

 const THIS_PATH =
  'src/platform/packages/shared/kbn-test/src/functional_test_runner/lib/config/run_check_ftr_configs_cli.ts';

+const IGNORED_FOLDERS = ['.buildkite/'];


this check was getting false positive on the newly added get_tests_from_config.ts

delanni · 2025-11-13T16:30:18Z

 set -euo pipefail

 source .buildkite/scripts/common/util.sh
+source .buildkite/scripts/bootstrap.sh


this is needed, so that the jest resolver can interpret our own plugins (such as @kbn/test)

delanni · 2025-11-13T16:32:03Z

  enabled?: Array<string | { [configPath: string]: { queue: string } }>;
 }

-function getEnabledFtrConfigs(patterns?: string[], solutions?: string[]) {


These functions are just moved to the bottom of the file, so the main function is first

delanni · 2025-11-13T16:32:23Z

+        ignore: [...DISABLED_JEST_CONFIGS, '**/node_modules/**'],
      })
    : [];
+  const jestUnitConfigs = await filterEmptyJestConfigs(


this is the core of the change, the rest is just readability refactors

delanni · 2025-11-13T16:33:35Z

@@ -0,0 +1,79 @@
+/*


@elastic/appex-qa - moved this to another file, not to bloat that already big file, most of the functionality doesn't overlap, so I think it's cleaner this way

Copilot

Pull Request Overview

This PR fixes an issue where Jest unit test grouping via ci-stats became inefficient after empty test configs were excluded from execution. The main change filters out empty Jest configs before requesting grouping, which reduces parallel test groups from ~53 back to the optimal ~23.

Key changes:

Introduced filterEmptyJestConfigs function that uses Jest's internal APIs to detect configs with no test files
Moved shared utility functions (getRequiredEnv, runBatchedPromises) to common location
Refactored Scout test grouping logic into dedicated module for better separation of concerns

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`.buildkite/scripts/steps/test/pick_test_group_run_order.sh`	Added bootstrap step to support Jest internal utilities resolution
`.buildkite/scripts/steps/test/scout_test_run_builder.ts`	Updated to use refactored Scout grouping function and replaced `process.cwd()` with `getKibanaDir()`
`.buildkite/pipeline-utils/utils.ts`	Added shared utility functions `getRequiredEnv` and `runBatchedPromises`
`.buildkite/pipeline-utils/scout/pick_scout_test_group_run_order.ts`	Extracted Scout test grouping logic from main file into dedicated module
`.buildkite/pipeline-utils/scout/index.ts`	Export file for Scout utilities
`.buildkite/pipeline-utils/index.ts`	Added Scout module export
`.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts`	Refactored to use `filterEmptyJestConfigs`, moved helper functions to end of file, updated imports
`.buildkite/pipeline-utils/ci-stats/index.ts`	Added export for new `get_tests_from_config` module
`.buildkite/pipeline-utils/ci-stats/get_tests_from_config.ts`	New file implementing empty Jest config detection using Jest internal APIs
`src/platform/packages/shared/kbn-test/src/functional_test_runner/lib/config/run_check_ftr_configs_cli.ts`	Added filtering for `.buildkite/` folders in config discovery
`src/platform/packages/shared/kbn-developer-toolbar/jest.config.js`	Fixed incorrect path reference from `private` to `shared`

csr

LGTM, thanks for looking into the test config grouping and the fix!

delanni · 2025-11-14T13:50:16Z

/ci

Dosant

👍

kibanamachine · 2025-11-14T15:48:20Z

Starting backport for target branches: 8.19, 9.1, 9.2

https://github.com/elastic/kibana/actions/runs/19369827128

elasticmachine · 2025-11-14T15:48:21Z

💚 Build Succeeded

Buildkite Build
Commit: 6738de0

Metrics [docs]

✅ unchanged

History

💔 Build #361389 failed 0a76f01
💔 Build #361082 failed 2d59570
💔 Build #361037 failed 4d5ebc4
💔 Build #361017 failed feaa83f
💛 Build #360624 was flaky 83fed9d
💔 Build #360536 failed a644715

kibanamachine · 2025-11-14T15:56:56Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts
❌	9.2	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 242440

Questions ?

Please refer to the Backport tool documentation

delanni · 2025-11-17T10:15:33Z

Skipping backport, as the cause wasn't backported

## Summary It seems the jest unit test grouping through `ci-stats` is broken. It's probably because we no longer run and report empty test configs (after elastic#236549 we skip empty configs), thus CI stats will no longer be able to have an accurate guess as to how long they will take. Calculating with the default guessed config runtimes results in weird buckets. We went from ~23 jest unit test groups to ~53 (again, probably because the empty configs, that would run in a few seconds are counted as if they take a few minutes + overhead). This PR updates the `pick_test_group_run_order` logic, such that we no longer ask grouping for the configs that don't contain any test files to run. The result reset the jest unit parallel group count back to 23. For this, I'm using jest's internal utilities to search for test files, and had to bootstrap the step, so that the resolution doesn't break. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary It seems the jest unit test grouping through `ci-stats` is broken. It's probably because we no longer run and report empty test configs (after elastic#236549 we skip empty configs), thus CI stats will no longer be able to have an accurate guess as to how long they will take. Calculating with the default guessed config runtimes results in weird buckets. We went from ~23 jest unit test groups to ~53 (again, probably because the empty configs, that would run in a few seconds are counted as if they take a few minutes + overhead). This PR updates the `pick_test_group_run_order` logic, such that we no longer ask grouping for the configs that don't contain any test files to run. The result reset the jest unit parallel group count back to 23. For this, I'm using jest's internal utilities to search for test files, and had to bootstrap the step, so that the resolution doesn't break. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 4889045) # Conflicts: # .buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts # .buildkite/pipeline-utils/scout/index.ts # .buildkite/pipeline-utils/scout/pick_scout_test_group_run_order.ts # .buildkite/pipelines/fips/fips_pipeline.ts # .buildkite/scripts/steps/test/pick_test_group_run_order.sh # src/platform/packages/shared/kbn-test-saml-auth/jest.config.js

dmlemeshko · 2026-04-14T16:02:03Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

wip: filter empty jest tests

92330cd

delanni force-pushed the filter-empty-jest-tests branch from 1ad38bc to 92330cd Compare November 10, 2025 16:45

delanni added 2 commits November 11, 2025 09:28

use relative path in require

6250ae2

use jest's tools for filtering empty configs

b2b8aa4

delanni force-pushed the filter-empty-jest-tests branch from bb6314d to b2b8aa4 Compare November 11, 2025 13:30

delanni added 4 commits November 11, 2025 14:39

try debugging output

8b82206

remove test snippet, and debug

bf85ce4

bootstrap in pick_test_groups in order to make @kbn/test available

e06ea79

exclude node_modules from globby finds - they weren't present before …

a644715

…because the step wasn't bootstrapped

delanni changed the title ~~wip: filter empty jest tests~~ [CI] Filter empty jest tests before grouping Nov 12, 2025

fix scout runner imports

83fed9d

delanni added release_note:skip Skip the PR/issue when compiling release notes backport:all-open Backport to all branches that could still receive a release Team:Operations Kibana-Operations Team labels Nov 12, 2025

delanni marked this pull request as ready for review November 12, 2025 15:30

delanni requested a review from a team as a code owner November 12, 2025 15:30

Ikuni17 reviewed Nov 12, 2025

View reviewed changes

Comment thread .buildkite/pipeline-utils/ci-stats/get_tests_from_config.ts Outdated

delanni added 3 commits November 13, 2025 15:00

minor code restructure

8136e48

Merge branch 'main' into filter-empty-jest-tests

15e0834

propagate jest config issues in pick_test_group_run_order's discovery…

feaa83f

…, fix bad jest config

delanni requested a review from a team as a code owner November 13, 2025 14:12

kibanamachine and others added 2 commits November 13, 2025 14:40

Changes from node scripts/eslint_all_files --no-cache --fix

4d5ebc4

skip checking .buildkite for ftr configs in ftr-validator quick-check

2d59570

delanni requested a review from a team as a code owner November 13, 2025 15:52

delanni enabled auto-merge (squash) November 13, 2025 16:28

delanni commented Nov 13, 2025

View reviewed changes

delanni requested a review from Copilot November 13, 2025 16:36

Copilot AI reviewed Nov 13, 2025

View reviewed changes

Comment thread .buildkite/pipeline-utils/ci-stats/get_tests_from_config.ts

Ikuni17 approved these changes Nov 14, 2025

View reviewed changes

csr approved these changes Nov 14, 2025

View reviewed changes

delanni disabled auto-merge November 14, 2025 13:04

Merge branch 'main' into filter-empty-jest-tests

0a76f01

delanni added the ci:beta-faster-pr-build label Nov 14, 2025

Dosant approved these changes Nov 14, 2025

View reviewed changes

delanni enabled auto-merge (squash) November 14, 2025 14:04

Merge branch 'main' into filter-empty-jest-tests

6738de0

delanni merged commit 4889045 into elastic:main Nov 14, 2025
12 checks passed

kibanamachine added the v9.3.0 label Nov 14, 2025

delanni added backport:skip This PR does not require backporting and removed backport:all-open Backport to all branches that could still receive a release labels Nov 17, 2025

delanni deleted the filter-empty-jest-tests branch November 17, 2025 10:15

dmlemeshko mentioned this pull request Apr 14, 2026

[9.2] [CI] Filter empty jest tests before grouping (#242440) #263133

Closed

delanni mentioned this pull request Apr 23, 2026

[CI] Replace Jest resolver with fast glob in filterEmptyJestConfigs #265188

Merged

Conversation

delanni commented Nov 10, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Nov 12, 2025

Uh oh!

Uh oh!

delanni Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

delanni Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

delanni Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

delanni Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

delanni Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

csr left a comment

Choose a reason for hiding this comment

Uh oh!

delanni commented Nov 14, 2025

Uh oh!

Dosant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Nov 14, 2025

Uh oh!

elasticmachine commented Nov 14, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

kibanamachine commented Nov 14, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

delanni commented Nov 17, 2025

Uh oh!

dmlemeshko commented Apr 14, 2026

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

delanni commented Nov 10, 2025 •

edited by kibanamachine

Loading