Skip to content

Start the rollout engines without blocking the caller - #1888

Open
fzyzcjy wants to merge 1 commit into
tom/refactor-miles/op7-1from
tom/refactor-miles/op7-4
Open

Start the rollout engines without blocking the caller#1888
fzyzcjy wants to merge 1 commit into
tom/refactor-miles/op7-1from
tom/refactor-miles/op7-4

Conversation

@fzyzcjy

@fzyzcjy fzyzcjy commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Part of #1837

@gemini-code-assist

gemini-code-assist Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@fzyzcjy fzyzcjy mentioned this pull request Jul 28, 2026
Open
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from f9b8fd0 to e1a85ce Compare July 28, 2026 13:31
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch 2 times, most recently from 29763ac to c9e3b7e Compare August 3, 2026 23:45
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from 2c49d90 to fd39325 Compare August 4, 2026 00:39
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from c9e3b7e to 2817005 Compare August 4, 2026 00:39
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from fd39325 to b1169c7 Compare August 4, 2026 06:58
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch 2 times, most recently from f374cec to 9ccc542 Compare August 6, 2026 07:03
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from b1169c7 to 0802de3 Compare August 6, 2026 07:03
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 9ccc542 to eb4c695 Compare August 9, 2026 23:33
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from 0802de3 to d013498 Compare August 10, 2026 06:08
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from eb4c695 to 55de553 Compare August 10, 2026 06:08
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from d013498 to b9370a6 Compare August 10, 2026 16:07
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch 2 times, most recently from 3480f66 to 45a6586 Compare August 10, 2026 16:34
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch 2 times, most recently from 0957c83 to a6eac6e Compare August 12, 2026 02:48
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 45a6586 to 2ec48ab Compare August 12, 2026 02:48
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from a6eac6e to ac9ec83 Compare August 12, 2026 14:31
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 2ec48ab to 60d4ae2 Compare August 12, 2026 14:31
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 60d4ae2 to 6bb2627 Compare August 12, 2026 23:56
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from 0dbfbae to e4239e6 Compare August 13, 2026 01:13
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 6bb2627 to 69f5f78 Compare August 13, 2026 01:13
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from e4239e6 to 93d272d Compare August 13, 2026 03:25
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch 2 times, most recently from 1afdee7 to 9f666a9 Compare August 13, 2026 06:01
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from 93d272d to 7f90422 Compare August 13, 2026 06:01
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 9f666a9 to 8d505ca Compare August 14, 2026 04:46
@guapisolo

guapisolo commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Non-blocking: #1889 made the node probes concurrent, but dynamic port allocation still calls blocking ray.get() from async _BaseActorManager.alloc_ports() through PortAllocator.alloc(); this remains present at #2176 and the checked #2649 tip. Could we make port allocation awaitable, or explicitly offload the blocking call, so the RayWorkerManager event loop cannot be stalled?

@fzyzcjy
Start the rollout engines without blocking the caller
114691b 
start_rollout_servers ran each model's startup through async_utils.run,
so the orchestrator process sat on a nested event loop and the models
started one after another. It becomes a coroutine that gathers every
model's cells at once, and InferenceController -- which can no longer
do that work in __init__ -- gains an async create() factory.

Concurrent models share one PortAllocator: two allocators probing the
same node independently could hand out the same free port before either
engine binds it.
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-1 branch from 67e42b2 to 469c083 Compare August 29, 2026 12:40
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op7-4 branch from 8d505ca to 114691b Compare August 29, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yueming-yuan yueming-yuan Awaiting requested review from yueming-yuan yueming-yuan is a code owner
@maocheng23 maocheng23 Awaiting requested review from maocheng23 maocheng23 is a code owner
@guapisolo guapisolo Awaiting requested review from guapisolo
@jybsuper jybsuper Awaiting requested review from jybsuper
@Zhichenzzz Zhichenzzz Awaiting requested review from Zhichenzzz
@Shi-Dong Shi-Dong Awaiting requested review from Shi-Dong
@yushengsu-thu yushengsu-thu Awaiting requested review from yushengsu-thu
@Rockdu Rockdu Awaiting requested review from Rockdu

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fzyzcjy @guapisolo