docs(evaluator): fix submit() packager, FilesetRef import, and leaked titles

[SandyChapman]

Summary

The Evaluator docs had drifted from the current nemo_evaluator plugin SDK contract, so several copy-paste examples failed before reaching execution. This PR realigns the runnable snippets and fixes two unrelated broken page titles found along the way.

What changed

• submit() requires a metric bundle packager. Added metric_bundle_packager=CloudpickleMetricBundlePackager() (and its import) to every runnable evaluator.submit(...) example across index, sdk-resources, and the metrics/* + tutorials/* pages. run() examples are untouched — run() does not take the packager.
• FilesetRef import fix. The LLM Judge tutorial imported FilesetRef from the non-existent nemo_evaluator_sdk.values; it now imports from nemo_evaluator.sdk (an ImportError at the top of the tutorial before).
• ModelRef documented. metrics/model-configuration.mdx now states that local run() requires an inline Model, while remote submit() also accepts a ModelRef (workspace/model-name).
• Stale validation script replaced. test_doc_examples.py no longer calls the old /v2/... REST endpoints; it now runs offline contract checks for the SDK import paths and the submit() packager requirement.
• Leaked title marker. Two tutorial pages rendered @nemo-nb: hide as the page title (a nemo_nb cell marker that leaked into frontmatter title). Set real titles and removed the duplicate body H1, matching every other evaluator page.

Not changed (investigated, no action needed)

• Doc links / slugs. The reported "stale slugs" are correct — Fern derives slugs from page titles, so agentic-metrics, rag-metrics, bring-your-own-metric, etc. resolve. make docs-broken-links passes.

Testing

• make docs-check → 0 errors, 194 MDX files parse cleanly
• make docs-broken-links → all checks passed
• ruff + ty clean; new contract test passes 7/7
• Both fixed tutorial titles confirmed via the Fern dev-server render (correct <title>/<h1>, no @nemo-nb)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Updated remote/durable job examples across evaluator docs and tutorials to consistently include metric_bundle_packager=CloudpickleMetricBundlePackager() when submitting jobs.
  • Expanded model configuration guidance to better explain local inline models vs remote ModelRef usage.
  • Refreshed tutorial code snippets for judge evaluations to include the metric bundle packager in remote submissions.
• Tests
  • Converted evaluator documentation checks from HTTP-based validation to offline contract verification, adding assertions around importability and submit-only metric_bundle_packager behavior.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-claude-pedantic-sinoussi-2c5cf8.docs.buildwithfern.com/nemo-platform

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	21176/27762	76.3%	61.2%
Integration Tests	12216/26531	46.0%	19.5%

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3f37ce5e-ecc1-4961-b08c-be25def47d8c

📥 Commits

Reviewing files that changed from the base of the PR and between 4fd7b7a and 1d156b8.

📒 Files selected for processing (2)

• docs/evaluator/metrics/model-configuration.mdx
• docs/evaluator/test_doc_examples.py

🚧 Files skipped from review as they are similar to previous changes (2)

• docs/evaluator/metrics/model-configuration.mdx
• docs/evaluator/test_doc_examples.py

---

📝 Walkthrough

Walkthrough

All evaluator.submit() documentation examples across evaluator metric pages, tutorials, and SDK reference docs are updated to import and pass metric_bundle_packager=CloudpickleMetricBundlePackager(). The offline contract test suite replaces the old HTTP-based test. Model configuration docs gain inline Model vs ModelRef guidance. Two tutorial page titles are corrected.

Changes

CloudpickleMetricBundlePackager wiring across docs and tests

Layer / File(s)	Summary
Offline SDK contract tests docs/evaluator/test_doc_examples.py	Old HTTP workspace/job tests removed. New pytest functions validate CloudpickleMetricBundlePackager import path, FilesetRef/ModelRef module locations, metric_bundle_packager presence in Evaluator.submit vs absence in Evaluator.run, and ValueError when packager is omitted from submit. main() delegates to pytest.
Inline Model vs ModelRef guidance docs/evaluator/metrics/model-configuration.mdx	New subsections clarify that evaluator.run() requires a resolved inline Model, while evaluator.submit() accepts a ModelRef resolved at runtime; ModelRef is explicitly invalid for evaluator.run().
Packager in index and sdk-resources docs/evaluator/index.mdx, docs/evaluator/sdk-resources.mdx	Sync and async evaluator.submit() examples import CloudpickleMetricBundlePackager and pass it via metric_bundle_packager=.
Packager in all metric doc pages docs/evaluator/metrics/job-management.mdx, docs/evaluator/metrics/manage-metrics.mdx, docs/evaluator/metrics/results.mdx, docs/evaluator/metrics/llm-as-a-judge.mdx, docs/evaluator/metrics/remote.mdx, docs/evaluator/metrics/agent-configuration.mdx, docs/evaluator/metrics/rag.mdx, docs/evaluator/metrics/agentic.mdx, docs/evaluator/metrics/similarity.mdx	Every evaluator.submit() code block in these pages adds the CloudpickleMetricBundlePackager import and metric_bundle_packager= argument.
Tutorial titles and packager wiring docs/evaluator/tutorials/define-run-custom-python-metrics.mdx, docs/evaluator/tutorials/run-llm-judge-evaluation.mdx	Frontmatter titles corrected. LLM judge tutorial adds FilesetRef to SDK imports and wires CloudpickleMetricBundlePackager into both Prompt V1 and Prompt V2 evaluator.submit() calls.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#128: Enforces metric_bundle_packager as required for remote evaluate/submit paths—directly the SDK contract this PR documents and tests.
• NVIDIA-NeMo/nemo-platform#212: Adds runtime compilation tests for cloudpickle example specs, related to the same CloudpickleMetricBundlePackager submit path validated here.

Suggested labels

fix

Suggested reviewers

• arpitsardhana

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Title accurately summarizes the primary changes: adding required packager parameter to documentation examples, fixing incorrect imports, and correcting leaked frontmatter markers.
Docstring Coverage	✅ Passed	Docstring coverage is 88.89% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/pedantic-sinoussi-2c5cf8

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/evaluator/metrics/model-configuration.mdx`:
- Around line 213-218: The ModelRef instantiation in the evaluator.submit() call
is missing the required params argument. Add a params parameter to the ModelRef
constructor that specifies RunConfigOnlineModel configuration with the necessary
required parameters. This ensures the example complies with the upstream
contract that enforces RunConfigOnlineModel params for ModelRef targets and
prevents the TypeError at runtime.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d1a84a3d-0800-4a10-9f00-1cbbb34d8806

📥 Commits

Reviewing files that changed from the base of the PR and between 14077b9 and 4fd7b7a.

📒 Files selected for processing (15)

• docs/evaluator/index.mdx
• docs/evaluator/metrics/agent-configuration.mdx
• docs/evaluator/metrics/agentic.mdx
• docs/evaluator/metrics/job-management.mdx
• docs/evaluator/metrics/llm-as-a-judge.mdx
• docs/evaluator/metrics/manage-metrics.mdx
• docs/evaluator/metrics/model-configuration.mdx
• docs/evaluator/metrics/rag.mdx
• docs/evaluator/metrics/remote.mdx
• docs/evaluator/metrics/results.mdx
• docs/evaluator/metrics/similarity.mdx
• docs/evaluator/sdk-resources.mdx
• docs/evaluator/test_doc_examples.py
• docs/evaluator/tutorials/define-run-custom-python-metrics.mdx
• docs/evaluator/tutorials/run-llm-judge-evaluation.mdx