docs(evaluator-sdk): add Fabric harness config examples - #571
Merged
Conversation
Contributor
📝 WalkthroughWalkthroughAdds a new example module that defines Codex CLI and Hermes SDK harness configs, builds runtimes from a named harness, and prints each config as JSON. Updates ChangesFabric harness runtime example
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py`:
- Around line 8-9: The docstring in fabric_harness_runtimes.py has a
broken/incomplete sentence describing harness differences; update the
surrounding text so the clause introduced by adapter_id, runtime.transport, and
harness.settings is grammatically complete. Edit the docstring near the existing
configuration explanation in the module-level text so it reads as a finished
sentence rather than a cut-off fragment.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py`:
- Around line 72-74: The module docstring in FabricRuntime has an incomplete
sentence describing how harness config differs, so update that prose to match
the example module’s grammar and make the “config shape differs mainly in that
…” statement a complete sentence. Locate the text in the Fabric runtime
docstring near the mention of adapter_id, runtime.transport, and
harness.settings, and rewrite it so the harness selection/configuration
description reads cleanly and ends properly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0fc61336-4201-472e-846a-705850f18bb9
⛔ Files ignored due to path filters (1)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.pyis excluded by!sdk/**
📒 Files selected for processing (2)
packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
Contributor
|
Add examples/fabric_harness_runtimes.py showing Codex-CLI (subprocess, transport=cli) vs Hermes-SDK (in-library, transport=library) Fabric config shapes and building FabricAgentRuntime per harness, so callers can see how the structure differs (adapter_id, runtime.transport, harness.settings). The FabricAgentRuntime docstring now points at it. Addresses review feedback on #564. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
fabric-harness-examples/schapman
branch
from
July 6, 2026 16:06
5322542 to
1a20d4f
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py (1)
64-66: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
HARNESS_CONFIGS[harness]raises unhelpfulKeyErroron typo.🔧 Proposed fix
def build_runtime(harness: str, *, model: str | None = None, work_root: str | None = None) -> FabricAgentRuntime: """Build a :class:`FabricAgentRuntime` for a named harness (see :data:`HARNESS_CONFIGS`).""" + if harness not in HARNESS_CONFIGS: + raise ValueError(f"Unknown harness {harness!r}; choose from {sorted(HARNESS_CONFIGS)}") return FabricAgentRuntime(config=HARNESS_CONFIGS[harness], model=model, work_root=work_root)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py` around lines 64 - 66, The build_runtime helper currently indexes HARNESS_CONFIGS directly, so a typo in harness raises a bare KeyError with no context. Update build_runtime to validate the harness name before constructing FabricAgentRuntime, and raise a clearer exception that includes the invalid harness and the available harness names from HARNESS_CONFIGS. Use build_runtime and HARNESS_CONFIGS as the key symbols to locate the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py`:
- Around line 64-66: The build_runtime helper currently indexes HARNESS_CONFIGS
directly, so a typo in harness raises a bare KeyError with no context. Update
build_runtime to validate the harness name before constructing
FabricAgentRuntime, and raise a clearer exception that includes the invalid
harness and the available harness names from HARNESS_CONFIGS. Use build_runtime
and HARNESS_CONFIGS as the key symbols to locate the change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: eb8f0e1c-c361-4b9c-b7f1-33438e3b2925
⛔ Files ignored due to path filters (1)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.pyis excluded by!sdk/**
📒 Files selected for processing (2)
packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
✅ Files skipped from review due to trivial changes (1)
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
arpitsardhana
approved these changes
Jul 6, 2026
SandyChapman
enabled auto-merge
July 7, 2026 15:44
arpitsardhana
pushed a commit
that referenced
this pull request
Jul 9, 2026
Add examples/fabric_harness_runtimes.py showing Codex-CLI (subprocess, transport=cli) vs Hermes-SDK (in-library, transport=library) Fabric config shapes and building FabricAgentRuntime per harness, so callers can see how the structure differs (adapter_id, runtime.transport, harness.settings). The FabricAgentRuntime docstring now points at it. Addresses review feedback on #564. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py— a standalone example showing how the sameFabricAgentRuntimetargets different agent harnesses purely via the Fabricconfig, and points theFabricAgentRuntimedocstring at it.Why
Follow-up to #564 (Fabric agent-eval runner). Addresses @arpitsardhana's approval note: "It would be nice to add Fabric Profile example for different harness to get sense of different runtimes structure."
Contents
CODEX_CLI_CONFIG(nvidia.fabric.codex.cli) — subprocess transport (transport="cli"), codex-specificharness.settings.HERMES_SDK_CONFIG(nvidia.fabric.hermes.sdk) — in-library transport (transport="library"), explicit chat/message schemas.build_runtime(harness, model=…)helper and amain()that prints each config, so the structural difference (adapter_id,runtime.transport,harness.settings) is obvious.The module imports and builds the runtimes without the optional native deps (only
run_tasksneedsnemo-fabric/nemo-relay), so it stays importable in CI. Configs are grounded in Fabric's owntest_codex_cli.py/smoke_typed_config.py.Notes
Docs-only + a docstring pointer; vendored SDK mirror synced. No behavior change.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation