Skip to content

Drive the rollout server cells from reconcile against the worker provider - #2084

Open
fzyzcjy wants to merge 1 commit into
tom/refactor-miles/op13-76from
tom/refactor-miles/op13-77
Open

Drive the rollout server cells from reconcile against the worker provider#2084
fzyzcjy wants to merge 1 commit into
tom/refactor-miles/op13-76from
tom/refactor-miles/op13-77

Conversation

@fzyzcjy

@fzyzcjy fzyzcjy commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Part of #1837

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 212d060 to 82d7c81 Compare August 3, 2026 23:44
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch 3 times, most recently from 5da55c7 to cb206aa Compare August 4, 2026 02:36
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 1d0bcf9 to b04fd99 Compare August 4, 2026 06:55
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from cb206aa to 5fde492 Compare August 4, 2026 06:55
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from b04fd99 to 351e680 Compare August 4, 2026 07:27
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 5fde492 to 997e1d6 Compare August 4, 2026 07:27
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 351e680 to c341bdf Compare August 4, 2026 07:35
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 997e1d6 to 1c14c00 Compare August 4, 2026 07:35
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from c341bdf to a8b5a62 Compare August 6, 2026 03:48
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 1c14c00 to 24ac2ad Compare August 6, 2026 03:48
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from a8b5a62 to 25b1661 Compare August 6, 2026 07:04
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 24ac2ad to 09f4a5b Compare August 6, 2026 07:04
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 25b1661 to 806a1b8 Compare August 9, 2026 23:33
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 09f4a5b to 58e8ec9 Compare August 9, 2026 23:33
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 806a1b8 to d4617b7 Compare August 10, 2026 00:44
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 58e8ec9 to 16449f7 Compare August 10, 2026 00:44
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from d4617b7 to 8ec7c76 Compare August 10, 2026 06:05
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 16449f7 to 926a0a1 Compare August 10, 2026 06:05
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 8ec7c76 to a476326 Compare August 10, 2026 16:07
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 926a0a1 to 203b339 Compare August 10, 2026 16:07
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from a476326 to fa95826 Compare August 10, 2026 16:34
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch 2 times, most recently from 893b4b5 to f791417 Compare August 12, 2026 02:48
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch 2 times, most recently from eca669f to f056d2b Compare August 12, 2026 09:32
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch 6 times, most recently from d1c3208 to fbf0529 Compare August 13, 2026 03:11
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 7fed1a7 to e7c22a3 Compare August 13, 2026 03:25
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch 2 times, most recently from 499a2be to 77cf40a Compare August 13, 2026 06:00
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 5600293 to 9fa77b8 Compare August 13, 2026 06:39
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 77cf40a to 17b0691 Compare August 13, 2026 06:39
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-76 branch from 9fa77b8 to cdd197a Compare August 14, 2026 04:45
@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 17b0691 to 8c625a8 Compare August 14, 2026 04:46
@guapisolo

Copy link
Copy Markdown
Collaborator

[P2] Reject cross-node TP widths that do not tile the node

With num_gpus=24, num_gpus_per_engine=12, and num_gpus_per_node=8, this validation accepts the group and builds two one-worker/8-slot cells while each engine is launched with tp_size=12 and nnodes=1. Each worker therefore exposes only 8 GPUs to a 12-rank engine, so startup can fail to become healthy and wait indefinitely. No later PR through checkpoint #2176 or checked tip #2649 repairs this gap. Please fail fast unless partial-node workers are explicitly supported.

@fzyzcjy
fzyzcjy force-pushed the tom/refactor-miles/op13-77 branch from 8c625a8 to f5d8b8a Compare August 29, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants