Introducing --track-filter option

README.md

@@ -153,6 +153,55 @@ collected 58 items
 
 Logging shows that the plugin first listed all tracks and then generated test functions for each track-challenge combination it found.
 
+#### Filter tests based on tracks
+
+Marking classes or functions with @pytest.mark.track("track_name") like the following test class member 
+
+``` python
+class TestCustomParameters:
+    @pytest.mark.track("tsdb")
+    def test_tsdb_esql(self, es_cluster, rally):
+        ret = rally.race(
+            track="tsdb",
+            track_params={"run_esql_aggs": True, "index_mode": "time_series"},
+        )
+        assert ret == 0
+```
+can be used for the --track-filter option. If given that option with a comma-separated list of track-names
+pytest will generate tests only for those tracks. Example:  
+
+`pytest --log-cli-level=INFO --track-filter=big5,tsdb it/test_security.py it/test_custom_parameters.py`
+
+will skip the tests for security because they are marked to be using the 'elastic/security' track, while 
+testing custom parameters only contains a function marked to be using 'tsdb' track which is included in the 
+--track-filter option.
+
+```
+========================================================= test session starts ==========================================================
+platform darwin -- Python 3.12.8, pytest-7.1.2, pluggy-1.6.0 
+cachedir: .pytest_cache
+benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
+rootdir: <hidden>, configfile: pyproject.toml
+plugins: benchmark-3.4.1, httpserver-1.0.5, asyncio-0.19.0, rally-0.0.1, anyio-4.10.0
+asyncio: mode=Mode.STRICT
+collected 5 items                                                                                                                      
+
+it/test_security.py::TestSecurity::test_security_indexing SKIPPED (Skipping test for tracks ['elastic/security'] not in trac...) [ 20%]
+it/test_security.py::TestSecurity::test_security_indexing_querying SKIPPED (Skipping test for tracks ['elastic/security'] no...) [ 40%]
+it/test_security.py::TestSecurity::test_security_indexing_querying_logsdb SKIPPED (Skipping test for tracks ['elastic/securi...) [ 60%]
+it/test_security.py::TestSecurity::test_security_generate_alerts_source_events SKIPPED (Skipping test for tracks ['elastic/s...) [ 80%]
+it/test_custom_parameters.py::TestCustomParameters::test_tsdb_esql 
+------------------------------------------------------------ live log setup ------------------------------------------------------------
+INFO     pytest_rally.elasticsearch:elasticsearch.py:84 Installing Elasticsearch: [esrally install --quiet --http-port=19200 --node=rally-node --master-nodes=rally-node --car=4gheap,trial-license,x-pack-ml,lean-watermarks --seed-hosts="127.0.0.1:19300" --revision=current]
+INFO     pytest_rally.elasticsearch:elasticsearch.py:93 Starting Elasticsearch: [esrally start --runtime-jdk=bundled --installation-id=1ee3c852-b86c-4126-ad1e-ac2088e30335 --race-id=1d107ad1-28a0-4446-b71d-3787ae56fd19]
+------------------------------------------------------------ live log call -------------------------------------------------------------
+INFO     pytest_rally.rally:rally.py:144 Running command: [esrally race --track="tsdb" --track-repository="/Users/nikosdris/Projects/rally-tracks" --track-revision="improved-filtered-tests" --configuration-name="pytest" --enable-assertions --kill-running-processes --on-error="abort" --pipeline="benchmark-only" --target-hosts="127.0.0.1:19200" --test-mode --track-params="run_esql_aggs:True,index_mode:time_series"]
+PASSED                                                                                                                           [100%]
+---------------------------------------------------------- live log teardown -----------------------------------------------------------
+INFO     pytest_rally.rally:rally.py:91 Removing Rally config from [/Users/nikosdris/.rally/rally-pytest.ini]
+INFO     pytest_rally.elasticsearch:elasticsearch.py:104 Stopping Elasticsearch: [esrally stop --installation-id=1ee3c852-b86c-4126-ad1e-ac2088e30335]
+==================================================== 1 passed, 4 skipped in 40.22s =====================================================
+```
 #### Test execution
 
 Because our `test_autogenerated` function uses the [`es_cluster` fixture](#es_cluster), `pytest-rally` will install and start an Elasticsearch cluster during setup and stop it during teardown. All of our autogenerated tests will run their races with this cluster as their benchmark candidate.
@@ -301,3 +350,13 @@ The plugin includes the CLI option `--debug-rally`. If provided, the plugin will
 ## Skipping autogenerated tests
 
 The plugin [marks](https://docs.pytest.org/en/6.2.x/mark.html#mark) all autogenerated tests with `autogenerated`, a custom marker. If you would like to skip running tests generated by the plugin, simply pass `--skip-autogenerated-tests`. The plugin will then skip all tests with this marker. Note that this does not affect test collection.
+
+# Testing
+
+```
+python -mvenv .venv
+source .venv/bin/activate
+# provide full path to Rally source repository
+pip install -e "/<full-path>/rally[develop]"
+pytest
+```

pytest_rally/fixtures.py

@@ -44,3 +44,4 @@ def es_cluster(request, distribution_version, revision):
     cluster.start()
     yield cluster
     cluster.stop()
+

pytest_rally/plugin.py

@@ -52,6 +52,10 @@ def pytest_addoption(parser):
                     action="store_true",
                     default=False,
                     help=("If provided, Rally commands will just be logged, not executed."))
+    group.addoption("--track-filter",
+                    action="store",
+                    default="",
+                    help="Comma-separated list of track names to filter tests with (e.g., --track-filter=track1,track2)")
 
 @pytest.hookimpl
 def pytest_cmdline_main(config):
@@ -68,9 +72,15 @@ def current_branch(repo):
 
     repo = config.getoption("--track-repository", str(config.rootdir))
     rev = config.getoption("--track-revision", current_branch(repo))
-
+    tfilter = config.getoption("--track-filter")
+    if tfilter:
+        tfilter = [t.strip() for t in tfilter.split(",") if t.strip()]
+    else:
+        tfilter = []
+
     config.option.track_repository = repo
     config.option.track_revision = rev
+    config.option.track_filter = tfilter
 
 def validate_options(config):
     if config.option.distribution_version and config.option.revision:
@@ -99,6 +109,7 @@ def default_params(track, challenge):
 def pytest_generate_tests(metafunc):
     repo = metafunc.config.getoption('track_repository')
     rev = metafunc.config.getoption('track_revision')
+    tfilter = metafunc.config.getoption('track_filter')
     current_class = metafunc.cls
     desired_class = metafunc.config.option.test_class
 
@@ -108,7 +119,8 @@ def pytest_generate_tests(metafunc):
             params = []
             tracks_and_challenges = r.all_tracks_and_challenges()
             for track, challenges in tracks_and_challenges:
-                params += [(default_params(track, challenge)) for challenge in challenges]
+                if not tfilter or track in tfilter:
+                    params += [(default_params(track, challenge)) for challenge in challenges]
             metafunc.parametrize("track,challenge,rally_options", params)
             metafunc.definition.parent.add_marker("autogenerated")
 
@@ -118,3 +130,14 @@ def pytest_runtest_setup(item):
     if "autogenerated" in markers:
         if item.config.getoption("--skip-autogenerated-tests"):
             pytest.skip(msg="--skip-autogenerated-tests flag was set")
+
+    track_filter = item.config.getoption("track_filter")
+    if track_filter:
+        track_marker = item.get_closest_marker("track")
+        if track_marker:
+            # Support marker as a list: @pytest.mark.track(["track1", "track2"])
+            marker_tracks = track_marker.args[0]
+            if isinstance(marker_tracks, str):
+                marker_tracks = [marker_tracks]
+            if not any(track in track_filter for track in marker_tracks):
+                pytest.skip(f"Skipping test for tracks {marker_tracks} not in track_filter {track_filter}")

setup.py

@@ -20,7 +20,7 @@
 setup(
     name="pytest-rally",
     packages=["pytest_rally"],
-    version="0.0.1",
+    version="0.0.2",
     include_package_data=True,
     entry_points={"pytest11": ["name_of_plugin = pytest_rally.plugin"]},
     classifiers=["Framework :: Pytest"],

tests/conftest.py

@@ -50,7 +50,7 @@ def temp_repo(pytester, repo):
     temp_repo = pytester.mkdir("track-repo")
     copytree(repo, temp_repo, dirs_exist_ok=True)
     prefix = f"git -C {temp_repo}"
-    commands = ["init", "add .", "commit -am 'test'"]
+    commands = ["init -b main", "add .", "commit -am 'test'"]
     for command in commands:
         run_command_with_return_code(f"{prefix} {command}")
     yield temp_repo

tests/resources/track-repo/test-track2/index.json

@@ -0,0 +1,28 @@
+{
+  "settings": {
+    "index.number_of_replicas": 0
+  },
+  "mappings": {
+    "dynamic": "strict",
+    "properties": {
+      "geonameid": {
+        "type": "long"
+      },
+      "name": {
+        "type": "text"
+      },
+      "latitude": {
+        "type": "double"
+      },
+      "longitude": {
+        "type": "double"
+      },
+      "country_code": {
+        "type": "text"
+      },
+      "population": {
+        "type": "long"
+      }
+    }
+  }
+}

tests/resources/track-repo/test-track2/track.json

@@ -0,0 +1,109 @@
+{
+  "version": 2,
+  "description": "Skeleton track for testing pytest-rally",
+  "indices": [
+    {
+      "name": "test",
+      "body": "index.json"
+    }
+  ],
+  "corpora": [
+    {
+      "name": "test-track",
+      "documents": [
+        {
+          "source-file": "documents.json",
+          "document-count": 100,
+          "uncompressed-bytes": 100
+        }
+      ]
+    }
+  ],
+  "challenges": [
+    {
+      "name": "index-and-query",
+      "default": true,
+      "schedule": [
+        {
+          "operation": {
+            "operation-type": "delete-index"
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "create-index"
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "cluster-health",
+            "request-params": {
+              "wait_for_status": "green"
+            },
+            "retry-until-success": true
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "bulk",
+            "bulk-size": 5000
+          },
+          "warmup-time-period": 120,
+          "clients": 8
+        },
+        {
+          "operation": {
+            "operation-type": "force-merge"
+          }
+        },
+        {
+          "operation": {
+            "name": "query-match-all",
+            "operation-type": "search",
+            "body": {
+              "query": {
+                "match_all": {}
+              }
+            }
+          },
+          "clients": 8,
+          "warmup-iterations": 1000,
+          "iterations": 1000,
+          "target-throughput": 100
+        }
+      ]
+    },
+    {
+      "name": "index-only",
+      "schedule": [
+        {
+          "operation": {
+            "operation-type": "delete-index"
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "create-index"
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "cluster-health",
+            "request-params": {
+              "wait_for_status": "green"
+            },
+            "retry-until-success": true
+          }
+        },
+        {
+          "operation": {
+            "operation-type": "bulk",
+            "bulk-size": 5000
+          },
+          "warmup-time-period": 120,
+          "clients": 8
+        }
+      ]
+    }
+  ]
+}

tests/test_plugin.py

@@ -17,42 +17,59 @@
 
 import pytest
 
-def test_generates_tests_from_list_tracks(pytester, example, temp_repo):
-    expected = [
-        "test_track_challenge[test-track-index-and-query]",
-        "test_track_challenge[test-track-index-only]",
-    ]
-    generated, _ = pytester.inline_genitems(example, f"--track-repository={temp_repo}")
-    assert [func.name for func in generated] == expected
-
-def test_runs_correct_race_commands(caplog, temp_repo, run):
-    def expected_log_line(track, challenge):
-        command = (
-            f'esrally race --track="{track}" --challenge="{challenge}" '
-            f'--track-repository="{temp_repo}" --track-revision="main" '
-            '--configuration-name="pytest" --enable-assertions --kill-running-processes '
-            '--on-error="abort" --pipeline="benchmark-only" --target-hosts="127.0.0.1:19200" --test-mode'
-        )
+class TestPlugin:
+    # this should be sorted as per Rally list tracks output
+    tracks = ["test-track", "test-track2"]
+    challenges = ["index-and-query", "index-only"]
+
+    def test_generates_tests_from_list_tracks(self, pytester, example, temp_repo):
+        expected = [
+            f"test_track_challenge[{track}-{challenge}]" for track in self.tracks for challenge in self.challenges
+        ]
+        generated, _ = pytester.inline_genitems(example, f"--track-repository={temp_repo}")
+        assert [func.name for func in generated] == expected
+
+    def test_runs_correct_race_commands(self, caplog, temp_repo, run):
+        def expected_log_line(track, challenge):
+            command = (
+                f'esrally race --track="{track}" --challenge="{challenge}" '
+                f'--track-repository="{temp_repo}" --track-revision="main" '
+                '--configuration-name="pytest" --enable-assertions --kill-running-processes '
+                '--on-error="abort" --pipeline="benchmark-only" --target-hosts="127.0.0.1:19200" --test-mode'
+            )
+
+            return ("pytest_rally.rally", "INFO", f'Running command: [{command}]')
 
-        return ("pytest_rally.rally", "INFO", f'Running command: [{command}]')
-
-    challenges = [
-        "index-and-query",
-        "index-only",
-    ]
-
-    expected = [expected_log_line("test-track", challenge) for challenge in challenges]
-    res = run()
-    actual = [(r.name, r.levelname, r.message) for r in caplog.records if "esrally race" in r.message]
-    assert actual == expected
-
-def test_runs_correct_install_command(caplog, temp_repo, run):
-    expected = [
-        ("pytest_rally.elasticsearch", "DEBUG", 'Installing Elasticsearch: '
-         '[esrally install --quiet --http-port=19200 --node=rally-node --master-nodes=rally-node '
-         '--car=4gheap,trial-license,x-pack-ml,lean-watermarks --seed-hosts="127.0.0.1:19300" '
-         '--revision=current]')
-    ]
-    res = run()
-    actual = [(r.name, r.levelname, r.message) for r in caplog.records if "esrally install" in r.message]
-    assert actual == expected
+        challenges = [
+            "index-and-query",
+            "index-only",
+        ]
+
+        expected = [expected_log_line(track, challenge) for track in self.tracks for challenge in challenges]
+        res = run()
+        actual = [(r.name, r.levelname, r.message) for r in caplog.records if "esrally race" in r.message]
+        assert actual == expected
+
+    def test_runs_correct_install_command(self, caplog, temp_repo, run):
+        expected = [
+            ("pytest_rally.elasticsearch", "DEBUG", 'Installing Elasticsearch: '
+            '[esrally install --quiet --http-port=19200 --node=rally-node --master-nodes=rally-node '
+            '--car=4gheap,trial-license,x-pack-ml,lean-watermarks --seed-hosts="127.0.0.1:19300" '
+            '--revision=current]')
+        ]
+        res = run()
+        actual = [(r.name, r.levelname, r.message) for r in caplog.records if "esrally install" in r.message]
+        assert actual == expected
+
+    def test_track_filter_limits_tracks(self, pytester, example, temp_repo):
+        # Only "test-track2" should be included when filtered
+        generated, _ = pytester.inline_genitems(
+            example,
+            f"--track-repository={temp_repo}",
+            "--track-filter=test-track2"
+        )
+        expected = [
+            f"test_track_challenge[test-track2-index-and-query]",
+            f"test_track_challenge[test-track2-index-only]",
+        ]
+        assert [func.name for func in generated] == expected