[Research] Evaluate tool-description quarantine with prototype evidence

[mohanagy]

Summary

• add a research-only offline prototype for canonical MCP tool-descriptor drift and local prompt-injection heuristics
• publish the prototype method, synthetic results, option assessment, enforceable credential boundary, and a recommendation to defer runtime quarantine
• add contract coverage that keeps the evidence repeatable and prevents the research artifact from being described as a production control
• record the change under the Unreleased changelog

Evidence

The committed synthetic corpus intentionally exposes the limits of heuristic enforcement:

• 1/12 benign descriptions flagged (8.3%)
• 8/10 malicious descriptions flagged (80%); two misses
• 5/5 constructed descriptor changes detected by exact hashes
• 1,000-tool local hash-and-scan median 4.48 ms and p95 5.63 ms on Node 22.9.0 / Darwin arm64

These figures are explicitly not production false-positive estimates or startup SLOs. No design-partner evidence is claimed. The recommendation is to defer production implementation until representative partner workflows show value, acceptable approval churn, and a usable recovery path.

Validation

• npm exec -- vitest run tests/tool-description-quarantine-research.test.ts
• node scripts/tool-description-quarantine-prototype.mjs
• npm run lint
• npm run typecheck
• npm test -- --no-file-parallelism (146 files; 1,842 passed; 34 skipped)
• npm run build
• npm run check:pack (51 files)

Closes #36

Summary by CodeRabbit

• Documentation

  • Added research guidance for evaluating suspicious tool descriptions and descriptor changes.
  • Documented limitations, threat-model considerations, reproduction steps, and evidence requirements.
  • Clarified that the evaluation is research-only and does not provide runtime security protection.

• New Features

  • Added an opt-in command to run the tool-description evaluation prototype and generate results.

• Tests

  • Added coverage for detection results, change tracking, performance limits, and research-status documentation.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Added a research-only MCP tool-description quarantine prototype. It hashes and compares descriptors, scans descriptions, evaluates synthetic results, benchmarks execution, and exposes a repeatable npm command. Tests validate the report and documentation states that runtime quarantine is deferred.

Changes

Tool-description quarantine research

Layer / File(s)	Summary
Prototype evaluation pipeline scripts/tool-description-quarantine-prototype.mjs	The prototype scans descriptions, hashes and compares descriptors, validates descriptors, evaluates synthetic corpora, benchmarks 1,000 tools, and emits a JSON report.
Research execution and validation package.json, tests/tool-description-quarantine-research.test.ts	The npm script runs the prototype. Vitest validates report structure, deterministic metrics, descriptor validation, benchmark bounds, and research-only status.
Research and threat-model records docs/research/tool-description-quarantine.md, docs/threat-model.md, CHANGELOG.md	The documentation records the evaluation, limitations, evidence requirements, reproduction steps, threat-model boundaries, and changelog entry.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

A rabbit checks hashes in the hay,
Scans tool descriptions on display.
Tests guard each report,
Docs define the support,
Runtime quarantine waits another day.

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	❌ Error	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description includes Summary, Evidence, and Validation, but it omits the required Security impact section and several validation checklist confirmations.	Add a Security impact section covering credential handling and other listed boundaries, then confirm each required validation item or explain why it does not apply.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes provide prototype evidence, threat-boundary analysis, measurements, and a written recommendation that matches issue #36.
Out of Scope Changes check	✅ Passed	The prototype, documentation, tests, package script, threat-model reference, and changelog entry all support the research objectives in issue #36.
Title check	✅ Passed	The title clearly identifies the research evaluation and prototype evidence, which matches the pull request's main change.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch research/issue-36-tool-quarantine

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/research/tool-description-quarantine.md`:
- Line 33: Correct the run metadata line near “Run on 2026-08-11” by replacing
the future date with the actual date on which the research was executed, keeping
the recorded Node version and Darwin arm64 environment accurate.
- Line 80: Update the reproduction description near runPrototype() to accurately
list the fields emitted by the JSON report: aggregate corpus metrics, flagged
benign examples, misses, and aggregate benchmark values. Do not claim full
corpus observations or timing samples unless runPrototype() is changed to emit
them.

In `@scripts/tool-description-quarantine-prototype.mjs`:
- Around line 231-234: Update percentile to handle the 0.5 quantile correctly
for even-length sorted inputs by averaging the two middle values instead of
selecting only the upper middle value; preserve the existing behavior for
odd-length inputs, other quantiles, and empty arrays.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: dc06d7cb-8eec-4fee-b2b0-1096d3fe3875

📥 Commits

Reviewing files that changed from the base of the PR and between aa19264 and d52b00c.

📒 Files selected for processing (6)

• CHANGELOG.md
• docs/research/tool-description-quarantine.md
• docs/threat-model.md
• package.json
• scripts/tool-description-quarantine-prototype.mjs
• tests/tool-description-quarantine-research.test.ts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/tool-description-quarantine-prototype.mjs`:
- Around line 111-113: Update descriptorMap to validate every tool descriptor
has exactly one non-empty name and reject duplicate names before constructing
the Map. Ensure invalid descriptors produce an explicit invalid-descriptor
result or otherwise stop comparison, rather than allowing later entries to
overwrite earlier descriptors and letting compareDescriptors report no drift.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e8eebca9-ef48-451d-9a44-1666b8f0bdc2

📥 Commits

Reviewing files that changed from the base of the PR and between d52b00c and 55097b6.

📒 Files selected for processing (2)

• docs/research/tool-description-quarantine.md
• scripts/tool-description-quarantine-prototype.mjs

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/tool-description-quarantine-prototype.mjs`:
- Around line 95-98: Update the key comparator in the canonical descriptor
serialization chain around canonicalValue so sorting is locale-independent and
based on Unicode code-unit ordering instead of localeCompare(). Preserve
filtering undefined values and recursively canonicalizing each entry.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6b2351e9-ca94-48bc-97b3-bd92d9a2d229

📥 Commits

Reviewing files that changed from the base of the PR and between 55097b6 and 8c5447e.

📒 Files selected for processing (3)

• docs/research/tool-description-quarantine.md
• scripts/tool-description-quarantine-prototype.mjs
• tests/tool-description-quarantine-research.test.ts

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/research/tool-description-quarantine.md`:
- Around line 43-44: Reconcile the benchmark values in the documented
“1,000-tool hash-and-scan median” and “p95” entries with the PR summary: either
update the documentation to the same 4.59 ms median and 5.33 ms p95 values, or
clearly identify the recorded values as results from a separate run.
- Line 18: Revise the sentence describing Miftah’s descriptor comparison and
local decision requirement to present it as prototype behavior or a future
opt-in control, not an existing runtime requirement. Preserve the distinction
from the current strict cross-profile schema-mismatch guard and avoid implying
that changed snapshots are presently blocked pending operator approval.

In `@scripts/tool-description-quarantine-prototype.mjs`:
- Around line 9-30: Add the dotAll `s` flag to each regular expression in the
`scannerRules` array so `.` spans line terminators between matched terms, and
add a scanner test covering descriptor text with newline-separated terms for
multiline matching.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bbbe1b69-b142-49e1-a437-778539ef44db

📥 Commits

Reviewing files that changed from the base of the PR and between 8c5447e and 2e97cb2.

📒 Files selected for processing (2)

• docs/research/tool-description-quarantine.md
• scripts/tool-description-quarantine-prototype.mjs