docs: add model capability audit matrix

[HOYALIM]

Summary

• Adds a maintained model capability audit matrix page with result states, required evidence fields, scenario coverage, seed rows, and a completed-row template.
• Links the page from both OpenClaw and Hermes inference navigation.
• Adds a focused docs regression test so the matrix schema and nav links stay present.

Validation

• NODE_PATH=/Users/holim/code/NemoClaw/node_modules /Users/holim/code/NemoClaw/node_modules/.bin/vitest run test/model-capability-audit-doc.test.ts
• test/e2e/e2e-cloud-experimental/check-docs.sh --only-links --local-only docs/inference/model-capability-audit.mdx
• NODE_PATH=/Users/holim/code/NemoClaw/node_modules /Users/holim/code/NemoClaw/node_modules/.bin/tsx scripts/sync-agent-variant-docs.ts --check

DCO Verification

Signed-off-by: Ho Lim subhoya@gmail.com

Commits were rebased with sign-off trailers using git rebase --signoff origin/main.

Fixes #3123

Summary by CodeRabbit

Release Notes

• Documentation

  • Added a new Model Capability Audit page under Inference, including a maintained audit matrix template for both OpenClaw and Hermes variants.
  • Defined allowed audit states, required evidence fields, baseline scenario coverage, and added links for related artifacts and next steps.

• Tests

  • Added a documentation validation test to verify required audit state tokens, evidence-schema labels, the expected audit table headers, and both variant navigation links.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a new Model Capability Audit reference page, wires it into the Inference navigation for both documentation variants, and adds tests that verify the page’s required content and links.

Changes

Model Capability Audit documentation

Layer / File(s)	Summary
Audit Matrix MDX content docs/inference/model-capability-audit.mdx	Adds the page frontmatter, usage rules, allowed audit states, required evidence fields, scenario coverage, seeded matrix rows, completed-row template, related artifacts, and next steps links.
Navigation wiring and validation docs/index.yml, test/model-capability-audit-doc.test.ts	Adds model-capability-audit entries under Inference for both OpenClaw and Hermes, and adds Vitest checks for the audit states, evidence labels, table header, nav links, and next steps links.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• NVIDIA/NemoClaw#2823: Shares the same tool-calling reliability documentation referenced by the new audit page’s related links.

Suggested labels

feature

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the new model capability audit matrix docs.
Linked Issues check	✅ Passed	The PR adds the audit matrix doc, required schema and scenarios, OpenClaw/Hermes navigation, and regression tests aligned with #3123.
Out of Scope Changes check	✅ Passed	The changes stay focused on the audit matrix docs, nav links, and regression test with no unrelated code paths.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a “Model Capability Audit Matrix” documentation page and ensures it’s discoverable via the docs navigation, with a Vitest guard to keep the schema/states and nav links from drifting.

Changes:

• Add docs/inference/model-capability-audit.mdx defining result states, required schema, and seed matrix rows
• Link the audit page into both OpenClaw and Hermes guide variants in docs/index.yml
• Add a Vitest doc-audit test to enforce the expected states, evidence fields, and nav entries

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
test/model-capability-audit-doc.test.ts	Adds tests ensuring the audit doc contains required states/schema and is linked from both guide variants
docs/inference/model-capability-audit.mdx	Introduces the audit matrix doc, including required states, schema, scenarios, and seed rows
docs/index.yml	Adds nav entries pointing at the generated OpenClaw/Hermes audit pages

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

docs/inference/model-capability-audit.mdx (1)

112-115: ⚡ Quick win

Add a bottom “Next Steps” section with related page links.

The page ends with “Related Artifacts” but does not include the required “Next Steps” section that links to related docs pages.
As per coding guidelines, new pages should include “A ‘Next Steps’ section at the bottom links to related pages.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/inference/model-capability-audit.mdx` around lines 112 - 115, Add a
"Next Steps" section at the end of the model-capability-audit.mdx document after
the "Related Artifacts" section. This section should include links to related
documentation pages following the coding guidelines that require new pages to
have a "Next Steps" section at the bottom. The section should link to relevant
pages such as the model-specific-setup documentation and
tool-calling-reliability pages mentioned in the Related Artifacts to guide
readers on what to explore next.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/inference/model-capability-audit.mdx`:
- Around line 1-12: The frontmatter metadata at the top of the
model-capability-audit.mdx file is missing several required fields according to
the docs page-structure rules for new pages. Add the missing frontmatter fields:
topics, tags, difficulty, audience, and status to the existing metadata section
that already contains title, description, keywords, and content fields. These
fields should be added alongside the other frontmatter properties in the YAML
block at the beginning of the file to ensure compliance with the documentation
guidelines.

---

Nitpick comments:
In `@docs/inference/model-capability-audit.mdx`:
- Around line 112-115: Add a "Next Steps" section at the end of the
model-capability-audit.mdx document after the "Related Artifacts" section. This
section should include links to related documentation pages following the coding
guidelines that require new pages to have a "Next Steps" section at the bottom.
The section should link to relevant pages such as the model-specific-setup
documentation and tool-calling-reliability pages mentioned in the Related
Artifacts to guide readers on what to explore next.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b3243c81-958e-42d1-bf5b-82f47c4ebcec

📥 Commits

Reviewing files that changed from the base of the PR and between 1baae3c and 32fc504.

📒 Files selected for processing (3)

• docs/index.yml
• docs/inference/model-capability-audit.mdx
• test/model-capability-audit-doc.test.ts

[wscurran]

✨ Thanks for adding the model capability audit matrix page with result states, required evidence fields, scenario coverage, seed rows, and a completed-row template. This proposes a way to document model capabilities across agents with a maintained matrix and regression test to keep the schema and nav links present.

---

Related open issues:

• #3123 Model performance / capability audit across supported agents

[prekshivyas]

Comprehensive schema doc. Copilot's double-pipe assertion concern is stale — the table header is single-pipe and toContain works correctly against it. CodeRabbit's missing frontmatter finding is also stale — topics, tags, difficulty, audience, and status are all present in the current commit. exclude-from-skills-gen: true is the right call for a maintainer-facing reference. ## Next Steps section is present with correct cross-links.

[prekshivyas]

Retracting my approval — three categories of factual errors I missed on first pass:

Dead cross-links:

• "Related Artifacts" references nemoclaw-blueprint/model-specific-setup/README.md — that directory does not exist in the repo.
• "Next Steps" links to docs/inference/tool-calling-reliability — that page does not exist.

Wrong source file paths in all seed rows:
The doc cites src/lib/inference/config.ts, src/lib/inference/model-prompts.ts, src/lib/inference/local.ts, and src/lib/inference/vllm-models.ts — but there is no src/lib/inference/ subdirectory. The actual files are src/lib/inference-config.ts, src/lib/model-prompts.ts, and src/lib/local-inference.ts.

Please fix the cross-links and source citations before this merges.

[prekshivyas]

Correction to my previous comment: the source file paths and cross-links are all correct. I verified against a stale local checkout that was missing src/lib/inference/ and the nemoclaw-blueprint/model-specific-setup/ directory — both exist on origin/main. docs/inference/tool-calling-reliability also exists. No action needed on those items.

The original approval stands — the only earlier CodeRabbit/Copilot findings (frontmatter fields, test assertion) were already resolved in the current commit.

[prekshivyas]

Re-approving — previous retraction was based on a stale local checkout. All source paths, cross-links, and referenced directories verified correct against origin/main.

[prekshivyas]

Hi @HOYALIM — CI is green, just one thing needed before this can merge: your commits need DCO sign-offs.

Run this to add them in bulk:

git rebase --signoff origin/main
git push --force-with-lease

This adds a Signed-off-by: Ho Lim <subhoya@gmail.com> trailer to each of your commits, which certifies you have the right to submit this code under the project's license.

[miyoungc]

@HOYALIM Please also add DCO verification in the PR description.