Skip to content

feat(mcp): add config-gated lazy MCP schema loading - #33052

Closed
Interstellar-code wants to merge 1 commit into
NousResearch:mainfrom
Interstellar-code:upstream-mcp-lazy-phase1
Closed

feat(mcp): add config-gated lazy MCP schema loading#33052
Interstellar-code wants to merge 1 commit into
NousResearch:mainfrom
Interstellar-code:upstream-mcp-lazy-phase1

Conversation

@Interstellar-code

Copy link
Copy Markdown
Contributor

Refs #6839

Summary

This PR adds a narrow, config-gated Phase 1 implementation of lazy MCP schema loading to reduce per-turn tool-schema token overhead for MCP-heavy installs.

Scope in this PR:

  • replace full MCP tool schemas with lightweight stubs at request time
  • add a meta-tool to promote selected MCP tools to full schemas on demand
  • keep promoted-tool state per session
  • keep the feature off by default

This PR is intentionally limited to the core lazy-loading loop. It does not include later expansion ideas like server-level discovery, eager server promotion, or broader telemetry/reporting.

Problem

Large MCP installations inject hundreds of full tool schemas into every API call, even when the conversation does not need them. This creates large fixed prompt overhead and can block lower-TPM providers entirely.

Approach

When mcp.lazy_loading is enabled:

  • MCP tools are exposed to the model as request-time stubs
  • the model can call load_mcp_tools to request full schemas for selected tools
  • promoted tools remain full for the rest of the session
  • builtin/non-MCP tools remain unchanged

The hook is plugin-based and fail-open:

  • transform_tools rewrites the outgoing tool list just before API dispatch
  • on any internal failure, the original full tool list is used unchanged

Backward compatibility

  • off by default
  • no behavior change unless explicitly enabled

Included config keys

  • mcp.lazy_loading
  • mcp.lazy_stub_max_desc

Not included in this PR

  • server discovery / server stub mode
  • eager server promotion thresholds
  • telemetry/reporting extras
  • semantic/RAG preselection

Tests

Ran the narrowed Phase 1 suite:

  • tests/plugins/mcp_lazy/test_stubs.py
  • tests/plugins/mcp_lazy/test_pool.py
  • tests/plugins/mcp_lazy/test_promote_and_hook.py
  • tests/hermes_cli/test_config_validation.py

Command:

scripts/run_tests.sh tests/plugins/mcp_lazy/test_stubs.py tests/plugins/mcp_lazy/test_pool.py tests/plugins/mcp_lazy/test_promote_and_hook.py tests/hermes_cli/test_config_validation.py

Constraint: upstream already tracks broader lazy-loading variants in NousResearch#6839 and NousResearch#13332, so this PR is intentionally scoped to the smallest plugin-first Phase 1 slice: request-time stubs, explicit promotion, and per-session state only.
Rejected: Include baseline telemetry and reporting | expands scope beyond the core lazy-loading loop and adds unrelated review surface
Rejected: Include server-level discovery and eager-threshold behavior | better handled as a follow-up once the core stub/promotion loop lands
Confidence: high
Scope-risk: narrow
Directive: Keep future extensions (server discovery, eager promotion, telemetry) layered on top of this Phase 1 contract instead of broadening the initial API semantics.
Tested: scripts/run_tests.sh tests/plugins/mcp_lazy/test_stubs.py tests/plugins/mcp_lazy/test_pool.py tests/plugins/mcp_lazy/test_promote_and_hook.py tests/hermes_cli/test_config_validation.py
Not-tested: Full Hermes suite; live TPM/token measurements against a large MCP installation on upstream main
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused MCP-heavy-install proposal. This is an automated hermes-sweeper review; current main already ships the requested progressive-disclosure behavior.

  • 369075dc95bb998fdf493ef0f97dfa2d19c43d82 added Tool Search before this PR was opened, and it is contained in v2026.6.5.
  • model_tools.py:537 conditionally replaces MCP/plugin schemas with tool_search, tool_describe, and tool_call at the configured threshold.
  • tools/tool_search.py:632 returns a selected tool's full schema on demand; model_tools.py:1071 dispatches the selected underlying tool with session toolset scope enforcement.
  • The linked Feature: Lazy Tool Schema Loading — Two-Pass Tool Injection to Reduce Token Overhead #6839 discussion also records the maintained Tool Search direction.

Closing as implemented on main.

@teknium1 teknium1 closed this Jul 13, 2026
@teknium1 teknium1 added sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/mcp MCP client and OAuth type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants