feat(dynamo): support LoRA lifecycle

[biswapanda]

Summary

• discover Dynamo worker LoRA update endpoints
• pause all engines, load the filesystem adapter on every worker, wait for frontend visibility, then resume
• preserve the existing non-Dynamo LoRA update path

Dependency

• feat(inference): Dynamo integration with prime-rl #3176 adds Dynamo worker discovery

This branch includes the prerequisite commit because it targets main. Its effective diff will shrink when #3176 merges.

Test plan

• ruff check on the discovery, LoRA, and test files
• focused Dynamo discovery and LoRA lifecycle tests
• included in the aggregate branch validation: 37 focused tests passed

---

Note

Medium Risk
Changes the orchestrator weight-update and inference discovery paths for Dynamo and LoRA; misconfigured inference_world_size or partial worker capability can block startup or leave engines paused if resume fails (mitigated by finally resume in tests).

Overview
Adds Dynamo-based inference pool setup via dynamo_discovery_url and /v1/rl/workers, replacing static admin_base_url for worker admin endpoints and per-engine world_size. Discovery retries until the summed worker world_size matches weight_broadcast.inference_world_size, which is now required on the orchestrator when Dynamo is enabled and can be set on shared NCCL/NIXL/filesystem broadcast for runs without a local inference config.

DynamoInferencePool extends the static pool with discovered admin clients, optional per-worker update/load_lora system URLs, and readiness that also checks the frontend /v1/models. LoRA weight updates on Dynamo pause engines on admin clients, POST adapters to /v1/loras on every worker’s system endpoint, wait until the adapter appears on the frontend, then resume; non-LoRA updates still use the existing admin path. StaticInferencePool can inject pre-built admin clients for this flow.

Config validation blocks mixing Dynamo with admin_base_url or elastic discovery; RL resolution propagates weight_broadcast to the orchestrator and relaxes the single-node NCCL “two local GPUs” rule when inference_world_size is set for external inference.

^{Reviewed by Cursor Bugbot for commit 5ff9be2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5ff9be2. Configure here.}

[cursor]

Control clients omit auth headers

Medium Severity

_setup_control_clients builds worker admin and LoRA system clients without the API key or custom headers that setup_admin_clients always attaches. With VLLM_API_KEY or configured headers set, Dynamo pause/resume//v1/loras calls can fail auth while frontend readiness checks still succeed.

Additional Locations (1)

• src/prime_rl/utils/dynamo.py#L149-L156

 

^{Reviewed by Cursor Bugbot for commit 5ff9be2. Configure here.}

[cursor]

Discovered world sizes unused

Medium Severity

DynamoInferencePool stores per-worker _admin_world_sizes from discovery but never reads them. NCCL/NIXL init still derives rank offsets as inference_world_size // len(admin_clients), so heterogeneous Dynamo topologies (for example different prefill vs decode TP) get incorrect broadcast ranks.

Additional Locations (1)

• src/prime_rl/utils/client.py#L547-L578

 

^{Reviewed by Cursor Bugbot for commit 5ff9be2. Configure here.}