Skip to content

fix: MCP semantic filter regression — top_k not limiting tools #20322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shin-bot-litellm wants to merge 1 commit into from
Closed

fix: MCP semantic filter regression — top_k not limiting tools #20322

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions litellm/proxy/_experimental/mcp_server/semantic_tool_filter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,10 +172,20 @@ async def filter_tools(
if not query or not query.strip():
return available_tools

# Router should be built on startup - if not, something went wrong
# Lazy-build router from available_tools if not already initialized
if self.tool_router is None:
verbose_logger.warning("Router not initialized - was build_router_from_mcp_registry() called on startup?")
return available_tools
verbose_logger.info(
"Router not pre-built, building from available_tools on first call"
)
try:
self._build_router(available_tools)
except Exception as e:
verbose_logger.error(f"Failed to lazy-build router: {e}")
return available_tools

if self.tool_router is None:
verbose_logger.warning("Router still None after build attempt")
return available_tools
Comment on lines +175 to +188
Copy link
Contributor

@greptile-apps greptile-apps bot Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lazy router initialization happens in the critical request path on first call — creating SemanticRouter with embeddings can add significant latency. Consider pre-building the router during startup via build_router_from_mcp_registry() or add a startup check/warning if the router isn't initialized.

Context Used: Rule from dashboard - What: Avoid creating new database requests or Router objects in the critical request path.

Why: Cre... (source)

Prompt To Fix With AI 
This is a comment left during a code review.
Path: litellm/proxy/_experimental/mcp_server/semantic_tool_filter.py
Line: 175:188

Comment:
Lazy router initialization happens in the critical request path on first call — creating `SemanticRouter` with embeddings can add significant latency. Consider pre-building the router during startup via `build_router_from_mcp_registry()` or add a startup check/warning if the router isn't initialized.

**Context Used:** Rule from `dashboard` - What: Avoid creating new database requests or Router objects in the critical request path.

Why: Cre... ([source](https://app.greptile.com/review/custom-context?memory=0c2a17ad-5f29-423f-a48b-371852ac4169))

How can I resolve this? If you propose a fix, please make it concise.


# Run semantic filtering
try:
Expand Down
2 changes: 2 additions & 0 deletions litellm/proxy/hooks/mcp_semantic_filter/hook.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,8 @@ async def async_pre_call_hook(
tool_names_csv = self._get_tool_names_csv(filtered_tools)

_metadata_variable_name = self._get_metadata_variable_name(data)
if _metadata_variable_name not in data:
data[_metadata_variable_name] = {}
data[_metadata_variable_name]["litellm_semantic_filter_stats"] = filter_stats
data[_metadata_variable_name]["litellm_semantic_filter_tools"] = tool_names_csv

Expand Down
Loading