docs(evaluator-sdk): add Legal Agent Benchmark (LAB) examples - #814
Merged
Conversation
Contributor
|
SandyChapman
force-pushed
the
legal-agent-bench-examples/schapman
branch
2 times, most recently
from
August 7, 2026 12:47
884b30a to
d2daa91
Compare
Add two runnable examples showing how to run Harvey Labs' Legal Agent Benchmark (LAB) through NeMo Evaluator. - legal_agent_bench_fabric: task-driven path. Loads LAB's real rubric scorer + judge prompts from pinned source, runs a closed-book agent via Fabric (codex), scores workspace deliverables with LabRubricMetric, and supports offline re-scoring with an alternative judge (rescore.py). - legal_agent_bench_harbor: Harbor path. Generates LAB's Harbor task suite and runs it via the SDK's HarborAgentTaskRunner with an in-container rubric verifier, exposing per-criterion component scores. Judge credentials are supplied via env/CLI only. Imports that resolve only at runtime are suppressed with ty's own `# ty: ignore[unresolved-import]` (not mypy's `# type: ignore`, which ty does not honor): LAB's `evaluation.*` is downloaded by ensure_lab_source and never committed, and lab_verify.py's docx/openpyxl/markitdown are installed only in the Harbor task image, since that verifier runs inside the container. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
legal-agent-bench-examples/schapman
branch
from
August 7, 2026 15:00
d2daa91 to
fa20c7e
Compare
SandyChapman
marked this pull request as ready for review
August 7, 2026 15:01
Contributor
📝 WalkthroughWalkthroughAdds Fabric and Harbor examples for the Legal Agent Benchmark. The change includes pinned source preparation, task and suite generation, rubric judges, metrics, runtime runners, stored-trial rescoring, verifier outputs, tests, and usage documentation. ChangesFabric LAB workflow
Harbor LAB workflow
Sequence Diagram(s)sequenceDiagram
participant User
participant FabricRunner
participant HarborRunner
participant AgentEvaluator
participant LABJudge
User->>FabricRunner: run Fabric benchmark
FabricRunner->>AgentEvaluator: submit prepared taskset
AgentEvaluator->>LABJudge: score workspace deliverables
LABJudge-->>AgentEvaluator: return rubric results
User->>HarborRunner: run Harbor benchmark
HarborRunner->>AgentEvaluator: submit Harbor tasks
AgentEvaluator->>LABJudge: verify deliverables
LABJudge-->>AgentEvaluator: return reward and criterion scores
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 15
🧹 Nitpick comments (2)
packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md (1)
23-30: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
uv runin every Python command.
packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md#L23-L30: replacepython -mwithuv run python -m.packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md#L39-L45: replacepython -mwithuv run python -m.packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py#L38-L46: replacepython -mwithuv run python -m.As per coding guidelines, “Run Python scripts and tools through uv, such as
uv run <script>.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md` around lines 23 - 30, Update the Python command at packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md:23-30 to use uv run python -m; make the same replacement at README.md:39-45 and run_legal_agent_bench.py:38-46. Ensure every affected Python invocation runs through uv.Source: Coding guidelines
packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.py (1)
54-66: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPin the generated image inputs.
python:3.12-slimand every>=dependency can change between builds. The same generated suite can then use a different verifier environment or fail after a dependency release. Pin the base-image digest and tested package versions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.py` around lines 54 - 66, Update the generated _DOCKERFILE image definition to pin python:3.12-slim by a tested immutable digest, and replace every unbounded >= package constraint with the specific tested package version. Keep the existing package set and installation flow unchanged while ensuring repeated builds use identical inputs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/lab_rubric_metric.py`:
- Around line 67-78: Restrict the metric configuration in the constructor to
LAB’s fixed output directory: validate that output_subdir is exactly "output"
and reject other values, or remove the parameter and always use "output". Keep
_score_rubric and _judge initialization unchanged.
- Around line 275-288: Replace the manual brace-counting logic in the JSON
extraction loop with json.JSONDecoder.raw_decode starting at each candidate “{”.
Use the decoder’s returned end index to extract and return the first valid JSON
object, while continuing to the next candidate when decoding fails; ensure
braces inside quoted strings are handled correctly.
In
`@packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/prepare_lab_taskset.py`:
- Around line 254-288: Validate limit before loading the scorer and judge in the
task-building function: raise an appropriate error for negative values and
return an empty task list immediately when limit is zero. Preserve the existing
task filtering and positive-limit behavior in the loop after this early
validation.
In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/README.md`:
- Around line 17-25: Update README.md lines 17-25 to replace “vendors LAB's
evaluation/ module” with wording that the metric imports the downloaded, pinned
LAB source at runtime. Also update README.md line 55 to replace “vendored &
called” with the same runtime-import description; make no code changes.
- Around line 1-200: Refocus the README as a HOW-TO by moving the setup
prerequisites to the top, retaining only the steps needed to run and re-score
the benchmark. Move “Two design decisions,” “How LAB maps onto the native
model,” “Harnesses,” and “Gotchas” into linked Explanation or Reference pages,
and add a final “Next Steps” section linking to those pages and any relevant
follow-up documentation.
In
`@packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/test_lab_rubric_metric.py`:
- Around line 11-15: Update the documented commands to run through uv: in
packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/test_lab_rubric_metric.py
lines 11-15, use frozen uv run pytest for the targeted test; in
packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py
lines 28-37 and
packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py lines
19-26, use uv run -m for the runner and re-scoring commands respectively.
In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.py`:
- Around line 74-82: Update _render_deliverables to skip symlinked paths and
resolve each candidate before calling _extract_text. Only process files whose
resolved path remains within run_dir.resolve(), while preserving the existing
sorting, extraction, truncation, and formatting behavior.
- Around line 46-83: Update _extract_text and _render_deliverables to enforce
bounded input: skip or reject files exceeding a defined per-file size limit
before reading or converting them, and track a shared aggregate character budget
while rendering so the combined judge input cannot exceed it. Preserve
truncation behavior for files within the limits and stop adding content once the
aggregate budget is exhausted.
- Around line 32-43: Update _SYSTEM and _prompt so agent_output is clearly
delimited and explicitly treated only as untrusted evidence, never as
instructions to the grader. Keep task and rubric instructions authoritative, and
add adversarial tests covering deliverables that attempt to override the judge
or force a passing verdict.
In
`@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.py`:
- Around line 226-230: Remove the --judge-api-key argument from the CLI parser
and eliminate its propagation into generated task.toml verifier.env data. Keep
judge-base-url and judge-model handling unchanged, and rely on Harbor runtime
secret injection for JUDGE_API_KEY.
- Around line 223-224: Validate --limit consistently in prepare_lab_suite.py
lines 223-224 and run_legal_agent_bench.py lines 100-104, 120-123, and 180 by
enforcing a shared positive-value rule during argument parsing; reject zero and
negative values before task copying, reward-only selection, or component-mode
slicing.
- Around line 84-98: The ensure_lab_source function must validate cached LAB
content instead of trusting any existing tasks directory. During verified
archive extraction, store a manifest containing the pinned source identity and
expected task set, then validate that manifest and task contents before
returning source_root; reject mismatched or missing manifests and preserve the
existing download-disabled behavior.
In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md`:
- Around line 1-19: Refactor the README into a focused HOW-TO: place
prerequisites before the execution steps, use Sphinx substitutions for product
names, and provide both Python SDK and CLI workflows in tab sets. Move the
current file inventory, score schema, component details, and caveats into linked
reference or explanation pages, and finish with a “Next Steps” section linking
related documentation.
- Around line 40-45: Fix the command continuation in the README example by
removing the inline comment and trailing spaces after the backslash in the
`run_legal_agent_bench` invocation. Move the Harbor agent note above the command
or after its final line, preserving valid Bash line continuation.
- Around line 58-65: Update the fenced code block containing the Harbor
evaluation output example by declaring its language as text on the opening
fence. Preserve the example contents unchanged.
---
Nitpick comments:
In
`@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.py`:
- Around line 54-66: Update the generated _DOCKERFILE image definition to pin
python:3.12-slim by a tested immutable digest, and replace every unbounded >=
package constraint with the specific tested package version. Keep the existing
package set and installation flow unchanged while ensuring repeated builds use
identical inputs.
In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md`:
- Around line 23-30: Update the Python command at
packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md:23-30 to
use uv run python -m; make the same replacement at README.md:39-45 and
run_legal_agent_bench.py:38-46. Ensure every affected Python invocation runs
through uv.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f2e30a13-de76-462a-b693-44a4e1dd81a1
📒 Files selected for processing (11)
packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/README.mdpackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/lab_rubric_metric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/prepare_lab_taskset.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/test_lab_rubric_metric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.mdpackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_criteria_metric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py
arpitsardhana
approved these changes
Aug 7, 2026
Correctness and robustness fixes from the CodeRabbit review, scoped to what
a user running these examples would actually hit.
- LabRubricMetric now rejects output_subdir != "output". LAB's score_rubric
hardcodes run_dir/"output", so any other value graded nothing and reported
a false zero — the docstring already said it had to match, but nothing
enforced it.
- _parse_json uses json.JSONDecoder.raw_decode instead of counting braces,
which treated a "}" inside a string value as the end of the object and
errored the metric on responses like {"reasoning": "15 U.S.C. } sec 1"}.
- build_lab_tasks and the three --limit CLIs reject negative values and treat
0 as "no tasks". The loop capped after appending, so --limit 0 produced one
task and a negative limit sliced tasks off the end.
- lab_verify skips symlinks and out-of-tree paths before extraction. Path.is_file()
follows symlinks, so an agent could link /proc/self/environ — which holds
JUDGE_API_KEY in the verifier container — and have it sent to the judge. Also
bounds per-file size and total judge input, and delimits deliverables so the
judge treats agent-authored text as evidence rather than instructions.
- Harbor README: the run command's line continuation had trailing spaces and an
inline comment after the backslash, so copy-pasting it broke. Fenced blocks
now declare a language.
- Fabric README: LAB's evaluation/ module is imported at runtime from the pinned
download, not vendored into this repo.
Tests cover the output_subdir guard and the JSON parsing cases.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (2)
packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.py (2)
111-114: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winKeep agent-controlled filenames out of the judge prompt.
namecan contain newlines and prompt text. That text appears outside the protected deliverable body. Use a generated evidence identifier in the heading and markers.Proposed fix
- name = path.relative_to(run_dir).as_posix() + evidence_id = f"deliverable-{len(blocks) + 1}" # Fenced and labelled so the judge can tell deliverable text from its own instructions; the # system prompt tells it to treat everything in here as evidence, never as instructions. - blocks.append(f"## Agent Output: {name}\n<<<BEGIN DELIVERABLE {name}>>>\n{text}\n<<<END DELIVERABLE {name}>>>") + blocks.append( + f"## Agent Output: {evidence_id}\n" + f"<<<BEGIN DELIVERABLE {evidence_id}>>>\n{text}\n" + f"<<<END DELIVERABLE {evidence_id}>>>" + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.py` around lines 111 - 114, Update the block-building logic around name and blocks.append to avoid embedding the agent-controlled relative filename in the judge prompt. Generate a safe evidence identifier for each deliverable and use it in the heading and BEGIN/END markers, while retaining the filename only in trusted internal handling.
107-115: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep rendered text within both configured limits.
Lines 108 and 110 append truncation text after slicing. A truncated file can exceed
max_chars, and the full output can exceedmax_total_chars. Reserve space for the suffix before slicing.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.py` around lines 107 - 115, Update the text truncation logic before the `blocks.append` call to reserve the length of each truncation suffix when slicing, ensuring individual output remains within `max_chars` and cumulative output remains within `max_total_chars`/`budget`. Apply the same bounded approach to both truncation checks while preserving the existing suffixes and deliverable formatting.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.py`:
- Around line 111-114: Update the block-building logic around name and
blocks.append to avoid embedding the agent-controlled relative filename in the
judge prompt. Generate a safe evidence identifier for each deliverable and use
it in the heading and BEGIN/END markers, while retaining the filename only in
trusted internal handling.
- Around line 107-115: Update the text truncation logic before the
`blocks.append` call to reserve the length of each truncation suffix when
slicing, ensuring individual output remains within `max_chars` and cumulative
output remains within `max_total_chars`/`budget`. Apply the same bounded
approach to both truncation checks while preserving the existing suffixes and
deliverable formatting.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3b0ffa11-ffae-4052-b084-769533b0e6ad
📒 Files selected for processing (10)
packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/README.mdpackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/lab_rubric_metric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/prepare_lab_taskset.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/test_lab_rubric_metric.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.mdpackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/lab_verify.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.pypackages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py
🚧 Files skipped from review as they are similar to previous changes (8)
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/rescore.py
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/README.md
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/prepare_lab_suite.py
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/lab_rubric_metric.py
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/prepare_lab_taskset.py
- packages/nemo_evaluator_sdk/examples/legal_agent_bench_fabric/run_legal_agent_bench_fabric.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two runnable examples under
packages/nemo_evaluator_sdk/examples/showing how to run Harvey Labs' Legal Agent Benchmark (LAB) through NeMo Evaluator — the Evaluator analogue of NeMo Gym'slegal_agent_benchenvironment (a Harbor-native, rubric-judged benchmark).legal_agent_bench_fabric/— task-driven pathprepare_lab_taskset.py: downloads pinned LAB source and builds nativeAgentEvalTasks (documents → workspace seeds, skills → seeds + manuals, criteria → grader reference). Loads LAB's realevaluation.score_rubric+ judge prompt templates from source (faithful to the leaderboard scorer, not a re-implementation).lab_rubric_metric.py:LabRubricMetricgradesworkspacefilesystem evidence with an injected judge;OpenAICompatibleJudgeadapter (throttle + bounded retry/timeout + lenient JSON parse) so a namespaced NVIDIA judge works. Emits per-criterion verdicts as metric diagnostics.run_legal_agent_bench_fabric.py: closed-book agent via Fabric (codex default).rescore.py: offline re-scoring — re-grade a stored run bundle with a different judge, no agent re-run (reconstructs tasks from source, hydrates trials from the bundle). Demonstrates decoupled execution/scoring + pluggable judge.legal_agent_bench_harbor/— Harbor pathprepare_lab_suite.py: generates LAB's Harbor task suite (self-contained; port of Gym'sprepare.py, no Gym dep).run_legal_agent_bench.py: runs via the SDK'sHarborAgentTaskRunner,--mode reward|components.lab_verify.py(SDK-free in-container rubric verifier) +lab_criteria_metric.py(per-criterion component scores).Validation
harbor_reward+lab_criteria+ a blendedlegal_qualityview).Notes
rescore.pydepends on theread_trialsbundle loader (separate SDK PR); per-criterion diagnostics surface in the bundle once the evaluator diagnostics-plumbing change lands. Both are non-blocking for the rest of the examples.harbor,nemo-fabric,openai,anthropic) are installed out-of-band per each example's README (kept out of the SDK lock).Summary by CodeRabbit