Skip to content

[Bugfix][Mooncake] Fix DP worker bootstrap register ReadTimeout race #42199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
qingyuan18 wants to merge 1 commit into
base: main
Choose a base branch
from
+15 −3
Open

[Bugfix][Mooncake] Fix DP worker bootstrap register ReadTimeout race #42199

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -922,13 +922,25 @@ async def register_worker_with_bootstrap(self):
)
while True:
try:
async with httpx.AsyncClient() as client:
# timeout=30s: the bootstrap server runs inside the DP0 worker
# process and can stall the uvicorn event loop for several
# seconds while DP0 is holding the GIL (e.g. EFA MR
# registration of multi-GB KV caches across many NICs). The
# httpx default of 5s is not enough headroom and causes
# non-DP0 workers to hit ReadTimeout even though the server
# eventually responds 200 OK.
async with httpx.AsyncClient(timeout=30.0) as client:
response = await client.post(url, json=payload.model_dump())
response.raise_for_status()
logger.debug("Successfully registered with bootstrap server at %s", url)
break
except httpx.ConnectError:
# Bootstrap server not ready, wait for a while and retry.
except (
httpx.ConnectError,
httpx.TimeoutException,
httpx.RemoteProtocolError,
):
Comment on lines +937 to +941

@gemini-code-assist gemini-code-assist Bot May 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While adding httpx.TimeoutException and httpx.RemoteProtocolError addresses the specific ReadTimeout issue, it is recommended to catch the broader httpx.TransportError instead of just httpx.ConnectError. Transient network issues or a heavily loaded bootstrap server can manifest as httpx.ReadError or httpx.WriteError (both subclasses of TransportError), which would currently fall through to the except Exception block, potentially logging an empty error message and causing the same startup deadlock the PR aims to fix.

Suggested change
except (
httpx.ConnectError,
httpx.TimeoutException,
httpx.RemoteProtocolError,
):
except (
httpx.TransportError,
httpx.TimeoutException,
httpx.RemoteProtocolError,
):

# Bootstrap server not ready / event loop momentarily starved
# / connection dropped mid-flight. Wait for a while and retry.
await asyncio.sleep(1)
except Exception as e:
err_msg = (
Expand Down
Loading