fix(tests): group ClickHouse integration tests under xdist

[ironcommit]

Summary

• Fixes the evaluator publish-to-intake integration flake by running integration tests with pytest-xdist's loadgroup scheduler.
• Dynamically marks evaluator tests that depend on the ClickHouse fixture with xdist_group("nmp_intake_clickhouse"), so those tests stay on one worker.
• Keeps existing deployments integration groups working under loadgroup by adding their dynamic markers early enough for xdist to see them.
• Updates the root xdist scheduler hook to honor --dist, so unit/default runs can actually use loadscope while integration runs use loadgroup.

Root cause

The failing CI job hit a test-infrastructure race, not an application-code failure. The publish-to-intake tests use a session-scoped fixture that starts a fixed local ClickHouse container and removes it at fixture teardown.

With xdist, session scope is per worker, not global across all workers. If ClickHouse-backed tests are split across workers, each worker can independently start and tear down the same singleton container. That can produce startup conflicts or let one worker remove the container while another worker still needs it.

--dist loadgroup is the xdist mode that honors xdist_group markers, but it does not infer shared fixtures by itself. The test suite has to mark tests that share an external resource.

The repo also had a root pytest_xdist_make_scheduler hook that always returned LoadGroupScheduling, regardless of the requested --dist value. This patch teaches that hook to honor loadscope vs. loadgroup, so the Makefile split is real rather than just command-line text.

Fix

• Make PYTEST_DIST configurable in Makefile.
• Keep unit/default test runs on loadscope for normal fixture locality.
• Run test-integration and test-integration-ci with loadgroup.
• In evaluator integration collection, add xdist_group("nmp_intake_clickhouse") to any test whose fixture closure includes _clickhouse.
• Mark dynamic grouping hooks with tryfirst=True so the markers exist before xdist reads them for scheduling.
• Update root pytest_xdist_make_scheduler to return LoadGroupScheduling only for --dist loadgroup, LoadScopeScheduling for --dist loadscope, and otherwise fall through to xdist defaults.

Test plan

• make -n test-unit PYTEST_WORKERS=2 shows --dist loadscope.
• make -n test-integration PYTEST_WORKERS=2 shows --dist loadgroup.
• uv run --frozen ruff check conftest.py plugins/nemo-evaluator/tests/integration/conftest.py plugins/nemo-deployments/tests/integration/conftest.py
• uv run --frozen pytest packages/nmp_common/tests/observability/test_otel.py::TestLoggingIntegration::test_initialize_obs_configures_plain_logging_by_default -v --no-cov -n 2 --dist loadscope shows scheduling tests via LoadScopeScheduling.
• uv run --frozen pytest plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py -v --no-cov -n 2 --dist loadgroup shows scheduling tests via LoadGroupScheduling and groups tests under @nmp_intake_clickhouse.
• uv run --frozen pytest plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py::test_volume_lifecycle -v --no-cov -n 2 --dist loadgroup shows scheduling tests via LoadGroupScheduling and groups the test under @nemo_deployments_docker_integration.

The original failing commit touched unrelated frontend files; this PR addresses the pre-existing integration-test infrastructure flake exposed by that CI run.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The ClickHouse launcher and evaluator integration tests now support configurable endpoints, synchronized startup, TCP readiness checks, xdist grouping, worker-level readiness logging, and propagation of the resolved ClickHouse URL to Intake services.

Changes

ClickHouse integration test coordination

Layer / File(s)	Summary
Integration test distribution Makefile, plugins/nemo-deployments/tests/integration/conftest.py	Pytest distribution is configurable, integration targets use loadgroup, and the deployments collection hook is explicitly prioritized.
Configurable ClickHouse launcher services/intake/scripts/spans/run_clickhouse.sh	The launcher reads container, port, data, and lock settings from environment variables, coordinates startup with a lock, ensures directories, and logs readiness consistently.
Integration fixture endpoint wiring plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py	The fixture derives the endpoint from the configured URL, starts ClickHouse with explicit settings, waits for TCP readiness, reports the endpoint, cleans up the container, and passes the URL to Intake services.
Worker readiness reporting plugins/nemo-evaluator/tests/integration/conftest.py	ClickHouse tests are grouped on one xdist worker, and worker messages are emitted through the terminal reporter or stdout.

Sequence Diagram(s)

sequenceDiagram
  participant TestFixture
  participant ClickHouseScript
  participant Docker
  participant PytestWorkers
  participant IntakeServices
  TestFixture->>ClickHouseScript: start with configured endpoint
  ClickHouseScript->>Docker: lock, inspect, and start container
  Docker-->>ClickHouseScript: configured ports become available
  TestFixture->>TestFixture: wait for TCP readiness
  TestFixture->>PytestWorkers: report reachable endpoint
  TestFixture->>IntakeServices: pass NMP_INTAKE_CLICKHOUSE_URL

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#924: Updates the same ClickHouse launcher with dynamic image handling and image-mismatch validation.

Suggested labels: test

Suggested reviewers: mckornfield, mmogallapalli

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: grouping ClickHouse integration tests under xdist to avoid concurrent worker races.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/ci-job-89395904003

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@services/intake/scripts/spans/run_clickhouse.sh`:
- Around line 41-44: Restrict the conflict handling in the container startup
flow to cases where the specific ${container_name} is confirmed running after
docker run fails. Before proceeding to docker exec, recheck the container with
the existing docker-ps mechanism and only swallow the error when it is running;
propagate port-binding and other failures instead.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 61308ed5-777b-429f-94af-3d7ebaf0dacd

📥 Commits

Reviewing files that changed from the base of the PR and between f518e45 and 72b386a.

📒 Files selected for processing (1)

• services/intake/scripts/spans/run_clickhouse.sh

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	27599/35351	78.1%	62.5%
Integration Tests	16109/34069	47.3%	19.8%

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py (1)

66-66: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicated CLICKHOUSE_CI_LOG_KEY string constant across two files. Same literal independently declared in both; if one drifts, pytest_testnodedown silently drops messages with no error.

• plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py#L66: keep as the single source of truth (or move to a shared module) and import it from conftest.py.
• plugins/nemo-evaluator/tests/integration/conftest.py#L37: import CLICKHOUSE_CI_LOG_KEY from test_publish_to_intake.py (or shared module) instead of redeclaring the literal.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py` at line
66, Use a single source of truth for CLICKHOUSE_CI_LOG_KEY: retain the
definition in
plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py:66, and
update plugins/nemo-evaluator/tests/integration/conftest.py:37 to import and
reuse it instead of redeclaring the literal.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py`:
- Around line 151-172: Update the _clickhouse fixture so _file_lock protects
only the startup subprocess.run call, allowing xdist workers to use the shared
container concurrently. Replace the unconditional docker rm -f teardown with
coordination that removes CLICKHOUSE_CONTAINER only after the final worker
releases the shared session, preserving the container for workers still using
it.

---

Nitpick comments:
In `@plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py`:
- Line 66: Use a single source of truth for CLICKHOUSE_CI_LOG_KEY: retain the
definition in
plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py:66, and
update plugins/nemo-evaluator/tests/integration/conftest.py:37 to import and
reuse it instead of redeclaring the literal.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fd8b601a-23b0-404e-a829-c8ed95a10e79

📥 Commits

Reviewing files that changed from the base of the PR and between 72b386a and 80b5937.

📒 Files selected for processing (3)

• plugins/nemo-evaluator/tests/integration/conftest.py
• plugins/nemo-evaluator/tests/integration/test_publish_to_intake.py
• services/intake/scripts/spans/run_clickhouse.sh

[tylersbray]

Just a nit at this point, no biggie. Thanks.