fix(evaluator): load globbed dataset files in a deterministic order

[SandyChapman]

Why

discover_files returned whatever order the filesystem's readdir handed back:

files = [f for f in base_path.rglob("*") if f.is_file()]   # pattern is None
files = list(base_path.glob(pattern))                       # glob pattern

Rows from those files are concatenated in that order, and their positions become row_index on the resulting scores. So the same fileset scored twice could pair a row's score with a different input row — anyone correlating scores back to their inputs gets silently wrong pairings. parallelism=1 doesn't help; this happens before scoring.

How it surfaced

e2e/test_evaluator_plugin.py::test_fileset_fragment_and_glob_datasets was failing intermittently in Kind CPU e2e:

assert [1.0, 0.0, 1.0] == [1.0, 1.0, 0.0]

Same multiset, different order. Globbing part-*.json over two files gives:

file order	rows	scores
part-a, part-b (sorted)	alpha, beta, gamma	[1.0, 0.0, 1.0]
part-b, part-a	gamma, alpha, beta	[1.0, 1.0, 0.0]

The test had pinned the second — the arbitrary order it happened to observe — so it passed or failed on readdir order. The test was the symptom; the loader was the bug.

After

Both branches sort. This follows what the codebase already does elsewhere for directory walks — agent_seeds.py:66, harbor_runtime.py:501, fabric/skills.py:492 — rather than introducing a new convention.

The docstring records why, since "sorted" reads as cosmetic tidiness unless you know row_index depends on it.

• e2e expectation moves to sorted order, with a comment naming where the order comes from.
• Two unit tests in TestDiscoverFiles pin the guarantee, one per branch, with files created back-to-front so a filesystem returning creation order fails them. Worth noting the neighbouring test_glob_pattern_discovers_files wraps both sides in sorted() — it deliberately avoided asserting order, so it could not have caught this.

Verification

• ruff clean, ty at the main baseline (9 warning-level diagnostics, unchanged).
• packages/nemo_evaluator_sdk/tests: 1440 passed.
• The e2e test only runs in the Kind CPU job, so its expectation change is verified by the unit tests and by the decode above rather than executed locally — CI confirms it.

Split out of #1069, which is where the flake was diagnosed; this is an independent behavior fix and doesn't depend on it.

Summary by CodeRabbit

• Bug Fixes

  • Ensured files discovered through directories and glob patterns are processed in consistent, sorted order.
  • Made dataset row ordering and associated scores deterministic across runs and environments.

• Tests

  • Added coverage for sorted file discovery, including files created in varying orders.
  • Updated evaluator expectations to reflect deterministic row ordering.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a0dfe8cd-ebae-42f0-a45d-3e9e9b292479

📥 Commits

Reviewing files that changed from the base of the PR and between 3a48a10 and a3e60b8.

⛔ Files ignored due to path filters (1)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/datasets/loader.py is excluded by !sdk/**

📒 Files selected for processing (3)

• e2e/test_evaluator_plugin.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py
• packages/nemo_evaluator_sdk/tests/execution/test_metric_execution.py

🚧 Files skipped from review as they are similar to previous changes (3)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py
• e2e/test_evaluator_plugin.py
• packages/nemo_evaluator_sdk/tests/execution/test_metric_execution.py

---

📝 Walkthrough

Walkthrough

Dataset file discovery now sorts recursive and glob-matched paths. Tests verify deterministic directory and glob ordering. The evaluator plugin test updates expected scores for sorted file-path order.

Changes

Deterministic dataset ordering

Layer / File(s)	Summary
Sort discovered dataset files packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/datasets/loader.py	discover_files sorts recursive and glob-matched paths. Documentation states that file order determines concatenated row and score indices.
Validate sorted row ordering packages/nemo_evaluator_sdk/tests/execution/test_metric_execution.py, e2e/test_evaluator_plugin.py	Tests verify lexicographic ordering for directory and glob discovery. The evaluator test expects scores in sorted file-path order.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#735: Covers related glob-dataset end-to-end ordering behavior.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the deterministic ordering fix for globbed evaluator dataset files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch deterministic-dataset-file-order/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	31441/40061	78.5%	63.0%
Integration Tests	18307/38013	48.2%	20.8%

[tylersbray]

Thanks!