feat(bench): add agent-runnable value benchmark harness (#5604)

[abhi-0906]

Summary

Adds a developer- and agent-runnable value benchmark under scripts/bench/, exposed as npm run bench. It emits nemoclaw.bench.v1 JSON and a concise Markdown report for live inference latency and trace-backed sandbox startup; request-path policy overhead remains explicitly unsupported until dedicated instrumentation exists.

Related Issue

Closes #5604

Changes

• Measures OpenAI-compatible inference round trips and reports min/median/p95/mean/max.
• Ingests validated production onboard traces for sandbox cold-start timing and sanitized comparison context.
• Requires HTTPS except for true loopback HTTP, rejects URL credentials and redirects, and allowlists API-key variables to OPENAI_API_KEY and NVIDIA_INFERENCE_API_KEY.
• Redacts query values and configured secrets, omits remote error bodies, rejects arbitrary HTTP 2xx responses, and replaces untrusted trace status text with fixed report-safe reasons.
• Documents prerequisites, usage, schema, interpretation, and agent workflow in scripts/bench/README.md.
• Adds unit and process-level CLI coverage for statistics, traces, endpoint and secret boundaries, output, exit status, and prerequisite failures.

Exact-head verification for e75419fe5dc80c007d854a8e00e95c7bdcee3350:

• npx vitest run test/bench/bench.test.ts test/bench/bench-cli.test.ts — 53/53 passed.
• npx vitest run --project cli src/lib/onboard/sandbox-readiness-tracing.test.ts src/lib/onboard/machine/phase-progress.test.ts — 35 passed, 1 expected skip.
• npm run typecheck, npm run build:cli, npm run typecheck:cli, npm run lint, and repository policy checks — passed; lint reports one unrelated existing warning.
• Production trace probe — sandbox and readiness timing ingested correctly after the latest main restack; policy overhead correctly remained unsupported because the current span measures setup rather than request-path shield overhead.
• PR Review / Advisor — exact-head GPT review found no required fixes, resolve/justify items, or in-scope improvements; prior secret-boundary findings are resolved.
• E2E / Advisor — no required or optional E2E runs or targets.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — exact-head advisor review covered endpoint confinement, API-key selection, output redaction, completion validation, and trace lineage/status/duration checks; see the PR Review / Advisor run above and the maintainer security approval.
• Non-success, skipped, or missing CI check accepted by maintainer — automatic advisor jobs skip fork PRs by design; trusted manual exact-head PR-review and E2E runs are linked above. GPT produced a clean final review; the parallel Nemotron job completed but its synthesis JSON was unusable. Fork-advisor follow-up: ci: auto-retrigger branch-validation E2E on every commit and run advisors on fork PRs via workflow_run #6145. The conditional docs-only job is not applicable to this code PR.

Verification

• PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub — 13/13 at exact head.
• Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes — not claimed.
• Targeted tests pass for changed behavior
• Full npm test passes (broad runtime changes only) — not required for this isolated developer benchmark; the exact-head GitHub CI shards provide broad coverage.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — not applicable; this is not a doc-only PR.
• Doc pages follow the style guide (doc changes only) — not applicable; only the benchmark README changed.
• New doc pages include SPDX header and frontmatter (new pages only) — not applicable; no documentation page was added.

---

Signed-off-by: Abhimanyu Kumar abhimanyukumar7290@gmail.com
Signed-off-by: Aaron Erickson aerickson@nvidia.com

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

A local benchmark harness is added under scripts/bench/ to measure inference round-trip latency, sandbox cold-start time, and policy-shield overhead, then emit JSON and Markdown reports. It includes shared utilities, a CLI runner, tests, documentation, and an npm bench script.

Changes

NemoClaw Value Benchmark Harness

Layer / File(s)	Summary
Benchmark schema, stats, and redaction scripts/bench/lib.ts	Defines benchmark schema constants, metric types, report structures, latency statistics, environment collection, URL redaction, secret scrubbing, and trace ingestion re-exports.
Trace validation and metric extraction scripts/bench/trace-ingest.ts	Validates onboard trace artifacts, reads sandbox and policy spans, and converts them into latency metrics with ok, unsupported, or error statuses.
Inference timing and report rendering scripts/bench/lib.ts	Builds chat-completions requests, measures live inference round trips, renders the Markdown benchmark report, and checks for blocking metric errors.
CLI parsing, report assembly, and output scripts/bench/run.ts, package.json	Defines CLI options and usage text, parses flags, resolves API keys from environment variables, reads trace artifacts, assembles the report, validates prerequisites, writes JSON and Markdown outputs, and sets process exit codes. Adds the npm bench script entry.
Tests and README test/bench/bench.test.ts, scripts/bench/README.md	Adds Vitest coverage for stats, URL helpers, redaction, inference timing, trace ingestion, markdown rendering, and blocking-error detection, plus README documentation for prerequisites, usage, outputs, and exit behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested reviewers: jyaunches, ericksoa, cjagwani

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.57% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR implements the requested benchmark harness, metrics, JSON/Markdown output, docs, secret handling, and clear unsupported/error behavior.
Out of Scope Changes check	✅ Passed	The changes stay focused on the benchmark workflow, tests, and documentation with no obvious unrelated additions.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding an agent-runnable benchmark harness under scripts/bench.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/bench/run.ts`:
- Around line 222-223: The error messages in the preflight checks for missing
options.baseUrl and options.model only mention the OPENAI_BASE_URL and
OPENAI_MODEL environment variables, but the argument parser also accepts
NEMOCLAW_BENCH_BASE_URL and NEMOCLAW_BENCH_MODEL. Update both error message
strings to include all supported environment variable names so users are aware
of all available options during setup failures.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1e9c6514-583c-4f42-a2ac-432d9f358db9

📥 Commits

Reviewing files that changed from the base of the PR and between a9f31e4 and 370217b.

📒 Files selected for processing (5)

• package.json
• scripts/bench/README.md
• scripts/bench/lib.ts
• scripts/bench/run.ts
• test/bench/bench.test.ts

[abhi-0906]

CodeRabbit feedback addressed (commit 5a078c8): the preflight failure message now lists all supported environment variables (NEMOCLAW_BENCH_BASE_URL / NEMOCLAW_BENCH_MODEL alongside OPENAI_BASE_URL / OPENAI_MODEL).

[wscurran]

✨ Thanks for the proposed benchmark harness that measures inference-round-trip, sandbox-cold-start, and policy-shield-overhead timings. This proposes a way to run npm run bench to emit machine-readable JSON and a concise Markdown value report for NemoClaw's first-use and inference-path performance.

---

Related open issues:

• #5604 Epic: Add agent-runnable benchmark harness and value report

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/bench/run.ts`:
- Around line 126-127: Avoid calling process.exit() immediately after writing
the help output in main() and the related error/report paths in
scripts/bench/run.ts; process.stdout.write/process.stderr.write can be async and
lose piped output. Update the help/error handling around the usage branch and
the final reporting logic to use synchronous writes where needed and set
process.exitCode instead of exiting directly, so the streams in main() can drain
cleanly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 26b96016-21be-44f2-b7b3-722110cfc1dd

📥 Commits

Reviewing files that changed from the base of the PR and between fd296cb and cbbed1b.

📒 Files selected for processing (4)

• package.json
• scripts/bench/README.md
• scripts/bench/lib.ts
• scripts/bench/run.ts

✅ Files skipped from review due to trivial changes (1)

• scripts/bench/README.md

🚧 Files skipped from review as they are similar to previous changes (2)

• package.json
• scripts/bench/lib.ts

[prekshivyas]

Review (maintainer skill) — security clean, but blocked on validation

Security: clean. This is dev-only benchmark tooling under scripts/bench/ with no production/sandbox surface. No child_process/exec/eval — the only outbound call is fetch to a developer-configured inference endpoint (OPENAI_BASE_URL/NEMOCLAW_BENCH_BASE_URL); reports scrub secret-shaped substrings and record only api_key_present: boolean; no new dependencies (adds a bench npm script only). No secret/injection/exec concerns.

Why I'm not approving yet — the merge bar isn't met: this is a fork PR, so both PR advisors and the E2E recommendation advisor were skipped (no secrets access), and only 4 basic checks ran — the new test/bench/bench.test.ts and the standard CLI test shards never executed. So the harness itself is unverified by CI.

To unblock: a maintainer should run the PR-review/E2E advisor workflow against this fork PR (advisor-equivalent coverage), and get the CLI test shards to run (the branch may need a refresh so the current CI workflows apply). Once the tests actually run green, this is a straightforward approve given the clean security surface.

[prekshivyas]

The markdown-links failure is not caused by this PR's changes — it's a stale-branch issue. The file .agents/skills/nemoclaw-maintainer-policies/references/triage-instructions.md was added to main after this branch was cut, so the link checker can't find it when running against the PR head.

Please rebase or merge main into your branch to pick up the new file and clear the failure:

git fetch upstream
git merge upstream/main
git push

[cv]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[cv]

Reviewed exact head 2f3b00d after main refresh. Required CI is green, installer pins validate against OpenShell 0.0.72, all commits are Verified with DCO, CodeRabbit has no unresolved findings, and the focused benchmark/security review passed.

[cv]

Re-approving exact head 78f0016 after the maintainer refresh. Strict HTTPS is now required for remote endpoints (literal loopback exceptions only), the prior plaintext credential-exposure risk is closed, 51/51 focused benchmark/CLI tests pass, the conditional scan and Biome pass, all 32 repository gates are green, CodeRabbit is clean, and DCO/GitHub verification pass for all commits.

[ericksoa]

Reviewed exact head e75419f after the final main restack. The endpoint, credential-selection, response-validation, report-redaction, and trace-integrity boundaries are closed; 53/53 focused benchmark tests and the readiness/progress regression suite pass. The full required matrix is green, all 13 commits are Verified with DCO, CodeRabbit is clean, the trusted PR advisor has no remaining findings, and the E2E advisor requires no runs or targets. Approving for v0.0.74.