Tests tracks selectively based on PR changes by NickDris · Pull Request #858 · elastic/rally-tracks

NickDris · 2025-09-17T06:20:04Z

Test will now use the --track-filter options from pytest-rally plugin in order to run tests only on selected tracks of the repo based on the changes introduced by the PRs.
Changes are tracked on the topmost level folders.

e.g. if big5/README.md was only changed in a PR, CI will now only run the test for big5 track in both stateful and serverless.

Note: This needs pytest-rally plugin updates #5 #7

Copilot

Pull Request Overview

This PR implements selective test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository.

Updates test command configurations to support dynamic track filtering arguments
Adds pytest track markers to associate test classes and methods with specific tracks
Implements CI workflow changes to detect changed tracks and pass them as filter arguments

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyproject.toml	Updates test scripts to accept arguments and adds track marker configuration
.github/workflows/ci.yml	Adds track filtering workflow and updates test execution to use filtered tracks
.github/scripts/track-filter.py	Creates script to generate track filters based on directory structure
.buildkite/it/run_serverless.sh	Implements track filtering for Buildkite serverless tests
it_serverless/test_logs.py	Adds track marker for elastic/logs track
it/test_synthetic_source.py	Adds track markers for tsdb and nyc_taxis tracks
it/test_security.py	Adds track marker for elastic/security track
it/test_custom_parameters.py	Adds track markers for tsdb track
it/logs/test_logs_unmapped.py	Adds track marker for elastic/logs track
it/logs/test_logs.py	Adds track marker for elastic/logs track

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

.github/workflows/ci.yml

.buildkite/it/run_serverless.sh

gbanasiak

This looks good, many thanks for tackling this. I think we also need to be able to specify files/paths that trigger full set of tests (e.g. .buildkite/**, pyproject.toml, .github/**) regardless of what specific top-level track folders were matched.

.github/workflows/ci.yml

it/logs/test_logs.py

.github/scripts/track-filter.py

- 'elastic/logs' and 'elastic/security' marks changed to 'elastic' - Fix name for unit tests to include the value of --track-filter

NickDris · 2025-09-22T14:14:45Z

@elasticmachine update branch

We decided to go with a simple plan, where pytest-rally will just run everything from elastic top-level track if the elastic top-level directory was changed, which keeps it simple.

.buildkite/it/serverless-pipeline.yml

.github/workflows/ci.yml

.buildkite/it/serverless-pipeline.yml

Run full CI If there are no file differences between HEAD and master branch

NickDris · 2025-10-06T14:36:34Z

I will be testing more things before backporting to the correct version

Implements test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository. - Updates test command configurations to support dynamic track filtering arguments - Adds pytest track markers to associate test classes and methods with specific tracks - Implements CI workflow changes to detect changed tracks and pass them as filter arguments - Defines RUN_FULL_CI_WHEN_CHANGED env variable which point to files that when changed full CI is executed.

* Tests tracks selectively based on PR changes (elastic#858) Implements test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository. - Updates test command configurations to support dynamic track filtering arguments - Adds pytest track markers to associate test classes and methods with specific tracks - Implements CI workflow changes to detect changed tracks and pass them as filter arguments - Defines RUN_FULL_CI_WHEN_CHANGED env variable which point to files that when changed full CI is executed. * CI builds ES revision from sources based on es-version file (elastic#875) - Github workflows now execute CI with the additional --revision option, effectively building Elasticsearch from sources using a revision SHA and according to [Rally --revision rules](https://esrally.readthedocs.io/en/stable/command_line_reference.html#revision) - es-version added to root directory which determine what is the value of the --revision argument that will be used in CI.

esbenchmachine · 2025-10-07T12:41:53Z

💔 All backports failed

Status	Branch	Result
❌	8.15	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 858

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

Implements test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository. - Updates test command configurations to support dynamic track filtering arguments - Adds pytest track markers to associate test classes and methods with specific tracks - Implements CI workflow changes to detect changed tracks and pass them as filter arguments - Defines RUN_FULL_CI_WHEN_CHANGED env variable which point to files that when changed full CI is executed.

Run full CI If there are no file differences between HEAD and master branch

esbenchmachine · 2025-10-07T20:11:51Z

💔 All backports failed

Status	Branch	Result
❌	9.2	An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

backport --pr 858

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

esbenchmachine · 2025-10-07T20:40:44Z

💔 All backports failed

Status	Branch	Result
❌	9.2	An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

backport --pr 858

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

esbenchmachine · 2025-10-08T11:20:14Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

Implements test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository. - Updates test command configurations to support dynamic track filtering arguments - Adds pytest track markers to associate test classes and methods with specific tracks - Implements CI workflow changes to detect changed tracks and pass them as filter arguments - Defines RUN_FULL_CI_WHEN_CHANGED env variable which point to files that when changed full CI is executed. (cherry picked from commit 714442d) Co-authored-by: Nick Dris <nick.dris@elastic.co>

Implements test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository. - Updates test command configurations to support dynamic track filtering arguments - Adds pytest track markers to associate test classes and methods with specific tracks - Implements CI workflow changes to detect changed tracks and pass them as filter arguments - Defines RUN_FULL_CI_WHEN_CHANGED env variable which point to files that when changed full CI is executed. (cherry picked from commit 714442d) # Conflicts: # it/logs/test_logs_unmapped.py # it/test_custom_parameters.py # pyproject.toml

gbanasiak · 2025-12-05T11:51:01Z

💚 All backports created successfully

Status	Branch	Result
✅	8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

Implements test execution based on PR changes by integrating pytest-rally plugin's --track-filter option. Tests will now only run for tracks affected by changes in the top-level directories of the repository. - Updates test command configurations to support dynamic track filtering arguments - Adds pytest track markers to associate test classes and methods with specific tracks - Implements CI workflow changes to detect changed tracks and pass them as filter arguments - Defines RUN_FULL_CI_WHEN_CHANGED env variable which point to files that when changed full CI is executed.

* Tests tracks selectively based on PR changes (#858) * CI builds ES revision from sources based on es-version file (#875) * Drop Python 3.9 and introduce Python 3.13 in CI (#877) * [ES-13188] Update GH macOS hosted runner image (#883) * Address pytest deprecations (#911) * CI determines Elasticsearch build arguments using unified job in gh workflow [stateful] (#925) * Fix an error where ci arguments were not applied properly (#928) * Fix an error where ci arguments were not applied properly * Test the change * Test failed retry with another fix * Previous fix failed retry * Revert es-version to current * Keep the indents if if/else * Add backport reminder and reorganize the backporting actions. (#889) * Rename IT folders (#938) * Fix pytest skip argument (#905) * Backport reminders - fix missing default values in workflow (#947) * Upgrade pip to 25.2 * Fix conflict in it_tracks * Update deprecated gh actions * Track changes for 8.19 * Fix test job which does not require the output of other jobs * Add backport reminder and reorganize the backporting actions. (#889) * Fix an error where ci arguments were not applied properly (#928) * Reduce filtering scope in CI workflow (#908) * Add pytest marker for cars * Address merge conflicts * Run serverless CI only on PRs targeting master (#859) * Accept pragma risks for joins esql * Make esql-full-text-functions a snapshot_only_challenge * Set index template for ingest_mode: data_stream (#849)

NickDris added 4 commits September 17, 2025 09:14

Selective tests based on pr changes

533e19e

Selective tests for serverless

1095fdd

Removed 'excluded folders' logic from track filtering job

ff97b04

Mark tracks

d93762a

NickDris requested a review from Copilot September 17, 2025 06:27

Copilot AI reviewed Sep 17, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

.buildkite/it/run_serverless.sh Outdated Show resolved Hide resolved

NickDris requested a review from a team September 17, 2025 06:28

gbanasiak reviewed Sep 19, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

it/logs/test_logs.py Show resolved Hide resolved

.github/scripts/track-filter.py Show resolved Hide resolved

NickDris added 6 commits September 19, 2025 17:45

Run full stateful CI for specific paths

635003e

Create --track-filter argument with one pass

1f47774

Track filter 'elastic'

2ce6294

- 'elastic/logs' and 'elastic/security' marks changed to 'elastic' - Fix name for unit tests to include the value of --track-filter

Make special handling for elastic tracks

011fb96

Add .buildkite to folders for full CI testing

7cd727f

Add full CI logic into serverless

389ea19

elasticmachine and others added 5 commits September 22, 2025 16:14

Merge branch 'master' into selective-tests

15c5ae0

Remove special handling of elastic folder.

c2149a9

We decided to go with a simple plan, where pytest-rally will just run everything from elastic top-level track if the elastic top-level directory was changed, which keeps it simple.

Temporal use of forked pytest-rally for tests

ff57355

Rever 'Temporal use of forked pytest-rally for tests'

be5e429

Fix track filter argument to serverless tests

bd301b5

NickDris requested a review from gbanasiak September 23, 2025 16:15

NickDris self-assigned this Sep 23, 2025

NickDris requested review from a team and removed request for a team September 25, 2025 14:21

gbanasiak reviewed Sep 30, 2025

View reviewed changes

.buildkite/it/serverless-pipeline.yml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

Run full ci only on proper file/folder change

971861e

NickDris requested a review from gbanasiak October 1, 2025 17:53

gbanasiak reviewed Oct 1, 2025

View reviewed changes

.buildkite/it/serverless-pipeline.yml Outdated Show resolved Hide resolved

NickDris added 2 commits October 2, 2025 17:58

Simplify env variable

2988693

Fix yaml identation for buildkite pipeline

158c508

NickDris added a commit that referenced this pull request Oct 6, 2025

Fix issue introduced by #858 (#876)

c212b82

Run full CI If there are no file differences between HEAD and master branch

NickDris added the v9.2 label Oct 7, 2025

NickDris added a commit that referenced this pull request Oct 7, 2025

Fix issue introduced by #858 (#876)

d76d306

Run full CI If there are no file differences between HEAD and master branch

NickDris added v9.2 and removed backport-to-8.15 v9.2 labels Oct 7, 2025

NickDris added v9.2 and removed v9.2 labels Oct 7, 2025

NickDris added v9.2 and removed v9.2 labels Oct 8, 2025

esbenchmachine mentioned this pull request Oct 8, 2025

[9.2] Tests tracks selectively based on PR changes (#858) #881

Merged

esbenchmachine mentioned this pull request Oct 8, 2025

Drop Python 3.9 and introduce Python 3.13 in CI #877

Merged

gbanasiak mentioned this pull request Dec 5, 2025

[8.15] Tests tracks selectively based on PR changes (#858) #946

Closed

Comments

Conversation

NickDris commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

gbanasiak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NickDris commented Sep 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NickDris commented Oct 6, 2025

Uh oh!

esbenchmachine commented Oct 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

esbenchmachine commented Oct 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

esbenchmachine commented Oct 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

esbenchmachine commented Oct 8, 2025

💚 All backports created successfully

Questions ?

Uh oh!

gbanasiak commented Dec 5, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NickDris commented Sep 17, 2025 •

edited

Loading