Skip to content

feat(inference): add Dynamo admin plane and NCCL transfer - #3481

Open
biswapanda wants to merge 10 commits into
PrimeIntellect-ai:mainfrom
biswapanda:feat/dynamo-admin-plane-main
Open

biswapanda wants to merge 10 commits into
PrimeIntellect-ai:mainfrom
biswapanda:feat/dynamo-admin-plane-main

Conversation

@biswapanda

@biswapanda biswapanda commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • reconstruct the feat(inference): add Dynamo admin plane #3475 Dynamo integration on current main using the AdminPlane abstraction merged by chore(inference): introduce admin plane abstraction #3472
  • discover and pin one protocol-v1 Dynamo inference worker, then create a direct administration client at its advertised origin
  • initialize Prime's custom NCCL group for one trainer GPU and one inference GPU and apply startup policy v0 plus later policy updates through /collective_rpc
  • preserve the small vLLM 0.28 skip_gather compatibility path required by the installed inference runtime
  • add a Linux-only optional dynamo dependency group pinned to ai-dynamo==1.4.2, leaving macOS and Apple Silicon without Dynamo runtime packages
  • permit Pydantic through 2.13.5 in temporary ai-dynamo-runtime==1.4.2 resolver metadata until a compatible runtime release is available
  • keep essential URL, allowlist, credential, response-bound, topology, collective-RPC, and NCCL failure checks at their runtime boundaries

This PR absorbs the NCCL work previously proposed in closed #3482 and the dependency work proposed in #3485. It deliberately supports the minimal text-only RL loop with one external inference GPU. General multi-rank discovery-to-trainer configuration, VLM training, routed-expert transfer, filesystem workers, NIXL, LoRA, and generic metrics changes remain outside this PR.

Validation

  • uv run pytest -q tests/unit/inference/test_dynamo.py tests/unit/test_configs.py -k dynamo: 53 passed, 137 deselected
  • Ruff and git diff --check passed
  • independent correctness and security reviews found no critical or high-severity issue
  • exact pushed head 6a4cc649c4dfc36a395f72e64e0bf7e56f20b9ca passed Kubernetes runtime nccl-adminplane-r28 in namespace bis-rl-3
  • trainer and orchestrator completed steps 1 through 5; all five batches were trainable 2/2 with reward 1.0 and 0% error, cancellation, and truncation
  • vLLM received six NCCL broadcasts for policies v0 through v5; two completed broadcast markers remained; post-policy generation through Dynamo returned a non-empty completion; both runner exit files were zero
  • the final log scan found no traceback, CUDA error, NCCL error, runtime error, or assertion error

The Kubernetes harness used a separately hashed runtime-only copy of math_env with network access allowed because the selected null agent with subprocess runtime cannot enforce the task network policy. The exact 2,200-path product source tree remained immutable and verified. The r28 config intentionally omitted weight_broadcast.inference_world_size, exercising the existing one-worker default after removal of the new public field.

@biswapanda
biswapanda force-pushed the feat/dynamo-admin-plane-main branch from e522c51 to 6896b46 Compare September 4, 2026 17:04
@biswapanda
biswapanda force-pushed the feat/dynamo-admin-plane-main branch from a533e4a to b821c7f Compare September 4, 2026 20:21
@biswapanda biswapanda changed the title feat(inference): add Dynamo admin plane feat(inference): add Dynamo admin plane and NCCL transfer Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant