Add concurrent tool execution with sequential flag

[strawgate]

Overview

Implements parallel tool execution for sampling with configurable concurrency. When an LLM returns multiple tool calls in a single response, they can now be executed concurrently for improved performance.

Key Features

• Backward compatible: Default behavior remains sequential execution
• Opt-in concurrency: Use tool_concurrency parameter to enable parallel execution
• Safety first: Tools can declare sequential=True to force sequential execution
• Bounded control: Supports unlimited or bounded parallel execution

Example

result = await context.sample(
    messages="Fetch weather for NYC and LA",
    tools=[fetch_weather],
    tool_concurrency=0,  # Unlimited parallel execution
)

Implementation

• Uses anyio.Semaphore for bounded concurrency
• Uses existing fastmcp.utilities.async_utils.gather() for structured concurrency
• Preserves result order regardless of execution order
• Comprehensive test coverage for all execution modes

Closes #3012

Generated with Claude Code

[strawgate]

@coderabbitai please review

[coderabbitai]

@strawgate: I'll review the changes in this PR which implements concurrent tool execution with sequential flag support.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Walkthrough

A new optional parameter tool_concurrency was added across the sampling API to control parallel execution of tool calls (None = sequential, 0 = unlimited, N > 0 = bounded). SamplingTool gains a sequential: bool field; if any tool is marked sequential the batch runs sequentially. execute_tools was extended to support sequential, unlimited parallel, and semaphore-bounded parallel execution. The tool_concurrency argument is threaded from context.sample()/sample_step() through implementation layers to execute_tools.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: adding concurrent tool execution with a sequential flag control mechanism.
Description check	✅ Passed	The description covers all required sections including overview, key features, example, and implementation details, with proper checklist completion and issue reference.
Linked Issues check	✅ Passed	All objectives from #3012 are met: concurrency control implemented with three modes (None/0/N), per-tool sequential flag added, parameter exposed through Context.sample/sample_step, and structured concurrency with order preservation.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with the linked issue requirements; no unrelated modifications to documentation, configuration, or unrelated features were introduced.
Docstring Coverage	✅ Passed	Docstring coverage is 90.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch claude/issue-3012-20260129-0140

---

No actionable comments were generated in the recent review. 🎉

🧹 Recent nitpick comments

docs/servers/sampling.mdx (1)

505-507: Minor inconsistency between sample() and sample_step() parameter docs.

The tool_concurrency description for ctx.sample() (Line 506) mentions the sequential=True override behavior, but the ctx.sample_step() version (Line 559) omits it. Since sample_step() also respects the sequential flag (it calls execute_tools the same way), consider adding the same note for consistency.

Proposed fix

     <ResponseField name="tool_concurrency" type="int | None" default="None">
-      Controls parallel execution of tools. `None` (default) for sequential, `0` for unlimited parallel, or a positive integer for bounded concurrency.
+      Controls parallel execution of tools. `None` (default) for sequential, `0` for unlimited parallel, or a positive integer for bounded concurrency. If any tool has `sequential=True`, all tools execute sequentially regardless.
     </ResponseField>

Also applies to: 557-560

src/fastmcp/server/sampling/run.py (2)

294-296: Consider using lazy %-style or extra-args logging instead of f-strings in logger.exception().

Lines 296 and 305 use f-string interpolation inside logger.exception(). While not a bug, the string is always formatted even if the log level is disabled. This is a minor performance nit for hot paths.

Proposed fix

-            logger.exception(f"Error calling sampling tool '{tool_use.name}'")
+            logger.exception("Error calling sampling tool '%s'", tool_use.name)

Also applies to: 303-305

---

324-335: When requires_sequential overrides an explicit tool_concurrency, consider logging a debug message.

If a caller explicitly passes tool_concurrency=0 but a tool has sequential=True, the request silently falls back to sequential. A debug log here would help developers understand why their concurrency setting isn't taking effect.

Proposed addition

     # Execute sequentially if required or if concurrency is None (default)
     if tool_concurrency is None or requires_sequential:
+        if requires_sequential and tool_concurrency is not None:
+            logger.debug(
+                "tool_concurrency=%s overridden to sequential because a tool "
+                "has sequential=True",
+                tool_concurrency,
+            )
         tool_results: list[ToolResultContent] = []
         for tool_use in tool_calls:

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

src/fastmcp/server/sampling/run.py (1)

311-317: Simplify the requires_sequential check — avoid constructing dummy SamplingTool objects.

The current code creates a full SamplingTool instance (with a lambda) just to get a default sequential=False for unknown tool names. This can be simplified.

♻️ Suggested simplification

-    requires_sequential = any(
-        tool_map.get(
-            tool_use.name, SamplingTool(name="", parameters={}, fn=lambda: None)
-        ).sequential
-        for tool_use in tool_calls
-    )
+    requires_sequential = any(
+        (tool := tool_map.get(tool_use.name)) is not None and tool.sequential
+        for tool_use in tool_calls
+    )