fix: honor tool_call_timeout so slow MCP tool calls stop hanging - #46
Merged
sklinglernv merged 1 commit intoJul 29, 2026
Merged
Conversation
MCPStreamableHTTPClient builds its httpx client without a timeout, so httpx's 5 second default governs the MCP read stream. The transport takes a pre-built client and has no timeout arguments of its own, so nothing else applies. When a tool takes longer, the server finishes the work and writes the reply to a stream httpx already dropped. Nothing raises, so the caller hangs. tool_call_timeout should have caught this at 60 seconds, but it was stored and never passed to ClientSession(read_timeout_seconds=...), so the documented timeout was dead and the real limit was an undocumented 5 seconds. Streamable-http now passes tool_call_timeout to httpx and keeps connect at 5 seconds, since establishing a connection is not a tool call. All three transports pass read_timeout_seconds to ClientSession. create_mcp_client and MCPManager.create_from_server accept tool_call_timeout, which create_from_server previously had no way to set despite being the documented entry point. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
Collaborator
|
LGTM, thanks a lot for the fix! |
callingmedic911
added a commit
to NVIDIA-NeMo/nemo-helix
that referenced
this pull request
Jul 29, 2026
nooa built its streamable-http transport with an httpx client that had no timeout, so httpx's 5 second default governed the MCP read stream and any tool call taking longer than that hung forever. The tau3 agent hit this on every turn, because the tau3-runtime sidecar runs a user-simulator LLM inside start_conversation and send_message_to_user. That is fixed upstream in NVIDIA-NeMo/labs-OO-Agents#46, so this drops the local mcp_timeout.py shim and goes back to MCPManager.create_from_server, which now takes tool_call_timeout directly. The agent still asks for 300 seconds: nooa's 60 second default would do today but leaves no room for a slower model. The pin is a commit rather than a tag because no release carries the fix yet. The lock's extra wheel rows are generated: no package version moves except nooa. Verified against a live sidecar that start_conversation now returns in 7.3s instead of hanging, and that a full Harbor trial completes with no harness error. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
AnuradhaKaruppiah
pushed a commit
to AnuradhaKaruppiah/nemo-platform
that referenced
this pull request
Aug 27, 2026
…IDIA-NeMo#935) * feat(experimentalist): make the benchmark harness suite-driven The runner hardcoded Terminal Bench's workspace and framework skill and assumed a package held exactly one domain, so a second suite could not be expressed as a manifest. A manifest now carries its own workspace and framework_skills, and may set task_id_prefix to scope a multi-domain package to one domain; coverage is then enforced against the matching subset. Skill directories resolve before the baseline evaluation, because otherwise a typo'd skill name surfaces hours of image builds later. A config may also name a user_simulator, which exports the OpenAI-style credentials that tau-style task environments and their NL-assertion judge read. Those are assigned rather than defaulted: a developer's ambient OPENAI_API_KEY paired with this gateway's base URL would 401 on every call. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * feat(experimentalist): add the tau3-bench banking suite and configs Scopes sierra-research/tau3-bench@1 to its 97 banking_knowledge tasks. The quality partition reuses the 41/28/28 split from optimization-datasets feat/tau2-other-domains, whose tau2-banking-knowledge-NNN names map onto the canonical Hub IDs; only task IDs are recorded here, never task content. The 6/3/3 fast partition draws solely from the 87 tasks whose reward_basis is pure database state, so smoke runs score without an LLM judge. Both partitions were validated against live Harbor Hub. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * feat(experimentalist): add the tau3 NOOA agent under test A NOOA CodeAct agent that reaches the tau2 domain tools over MCP from the task's tau3-runtime sidecar. AGENT-SPEC.md stays domain-generic because the policy arrives at runtime in the task instruction, and it is what the Experimentalist mutates. Three things the task images required: Installing from the committed lock uses the pip that python:3.12-slim already ships, since tau3 images carry neither uv nor the curl its installer wants. Terminal Bench's uv-upload machinery is unnecessary here because every tau3 task shares that one base image. Model credentials are handed over at exec time. A task's [environment].env only interpolates its docker-compose file, and upstream wires those variables into the sidecar; the main service declares no environment, so the agent had none. mcp_timeout.py replaces nooa 0.0.6's MCP transport, which is built without an httpx timeout and so inherits httpx's 5 second default. The sidecar runs the user simulator inside start_conversation and send_message_to_user, which take longer; the server completed the work and advanced its state, but the reply landed on an abandoned stream and nooa surfaced no timeout, so the agent hung indefinitely. Verified end to end on the fast test split: all three tasks complete with no harness error, one solved, for a 1/3 baseline the optimizer can improve on. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * docs(experimentalist): document the tau3 banking benchmark Turns the single-suite README into one that covers both suites: how a manifest carries its own workspace, framework skills, and optional domain prefix, the banking provenance and partition derivation, and the two-container topology with its memory and build-timeout consequences. Also records that upstream's tau3 Dockerfiles clone tau2-bench without a ref, so a cold layer cache can pick up newer commits than an earlier run saw. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * refactor(experimentalist): write partition entries relative to the task-id prefix Every ID in a domain-scoped package repeats the domain prefix, so listing 109 of them buried the part that actually differs and made the banking manifest 143 lines of near-identical strings. Partition entries are now names inside the suite's task_id_prefix, joined back on at load so the rest of the runner keeps working in canonical IDs. The banking manifest drops to 50 lines and its splits are legible at a glance. Suites without a prefix, meaning Terminal Bench, are unaffected. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * refactor(experimentalist): name benchmark configs after their suite A config carries the models and optimizer depth for one suite, but the original pair was named only for its partition. Once tau3-smoke.yaml and tau3-quality.yaml sat beside them, smoke.yaml and quality.yaml read as the generic pair rather than the Terminal Bench one. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * docs(experimentalist): trim the tau3 section to what a reader needs Drops the container topology, the memory and build-timeout guidance, and the note about upstream's unpinned tau2-bench clone. What remains mirrors the Terminal Bench section: package, hash, Hub record, partition derivation, and the one config field the suite introduces. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> * build(experimentalist): repin nooa to the upstream MCP timeout fix nooa built its streamable-http transport with an httpx client that had no timeout, so httpx's 5 second default governed the MCP read stream and any tool call taking longer than that hung forever. The tau3 agent hit this on every turn, because the tau3-runtime sidecar runs a user-simulator LLM inside start_conversation and send_message_to_user. That is fixed upstream in NVIDIA-NeMo/labs-OO-Agents#46, so this drops the local mcp_timeout.py shim and goes back to MCPManager.create_from_server, which now takes tool_call_timeout directly. The agent still asks for 300 seconds: nooa's 60 second default would do today but leaves no room for a slower model. The pin is a commit rather than a tag because no release carries the fix yet. The lock's extra wheel rows are generated: no package version moves except nooa. Verified against a live sidecar that start_conversation now returns in 7.3s instead of hanging, and that a full Harbor trial completes with no harness error. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> --------- Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MCPStreamableHTTPClientbuilds its httpx client without a timeout, so httpx's5 second default governs the MCP read stream. The transport takes a pre-built
client and has no timeout arguments of its own, so nothing else applies. When a
tool takes longer, the server finishes the work and writes the reply to a stream
httpx already dropped. Nothing raises, so the caller hangs.