diff --git a/docs/about/backends.md b/docs/about/backends.md index 34fb8e3e95e..c4e9bcfcdcb 100644 --- a/docs/about/backends.md +++ b/docs/about/backends.md @@ -18,3 +18,6 @@ NeMo RL supports multiple generation/rollout backends to accommodate different m For detailed information on backend selection, configuration, and examples, see the [Generation Backends documentation](../design-docs/generation.md). +See [Weight Refit: Choosing a Transport](../guides/refit.md) to select colocated +IPC, NCCL, sparse delta, or NIXL checkpoint-engine refit for the configured +training and generation backends. diff --git a/docs/design-docs/checkpoint-engines.md b/docs/design-docs/checkpoint-engines.md new file mode 100644 index 00000000000..008b81763e5 --- /dev/null +++ b/docs/design-docs/checkpoint-engines.md @@ -0,0 +1,373 @@ +# Checkpoint Engine Design + +Checkpoint engines are runtime refit transports for non-colocated generation. +They let GRPO move policy weights directly from policy workers to generation +workers without using the driver as a model-sized staging point. + +The first built-in backend is NIXL. The current implementation targets policy +workers refitting non-colocated vLLM generation workers. Colocated generation +still uses the existing IPC/HTTP refit paths, and non-colocated generation +without checkpoint engines still uses the existing NCCL collective path. + +The user-facing guide is [Checkpoint-Engine Refit](../guides/checkpoint-engine-refit.md). + +## Goals + +Checkpoint engines are designed to: + +- keep GRPO orchestration independent from the transfer backend +- stream weight batches instead of materializing a full model copy in the driver +- let backend implementations own their metadata, buffers, and peer setup +- allow additional transfer backends through a class-path plugin + +Checkpoint engines do not replace durable training checkpoints. They are used +only for the runtime weight update between policy and generation workers. + +## Control Flow + +The refit lifecycle is coordinated by `CheckpointEngineWeightSynchronizer`: + +1. Read `policy.generation.refit_transport` and its matching `refit_cfg` scope. +2. Resolve the configured bucket size from the smallest fixed GPU capacity + reported by policy and vLLM workers before transfer buffers are allocated. +3. Instantiate the backend on policy workers and vLLM internal workers. +4. Call `prepare()` and collect Ray-serializable metadata from every backend + instance. +5. Initialize policy and rollout peers with the combined metadata list. +6. Keep the backend initialized across refits for that synchronizer. +7. For each refit, ask policy workers to send weights through the backend. +8. Ask generation workers to receive batches, directly copy supported + destination-local expert shards, and pass remaining tensors through vLLM's + normal weight-loading path. +9. Call `shutdown()` to finalize backend state when the synchronizer is no + longer needed. + +Policy metadata appears first in the combined metadata list, followed by +generation metadata. Backends receive `train_world_size` and +`rollout_world_size` so they can interpret that list. + +## Configuration Contract + +Checkpoint-engine refit uses the same selector as other non-colocated vLLM +transports: + +```yaml +policy: + generation: + backend: vllm + colocated: + enabled: false + refit_transport: nixl + refit_cfg: + nixl: + update_weights_bucket_memory_ratio: 0.05 + device: cuda + release_after_refit: false + backend_name: UCX + # Optional, cluster-specific eight-rail tuning. + backend_init_params: + engine_config: MAX_RMA_RAILS=8 + device_list: "mlx5_0,mlx5_1,mlx5_2,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_8" +``` + +`refit_transport` can select: + +- `nixl`, which maps to + `nemo_rl.utils.checkpoint_engines.nixl:NIXLCheckpointEngine` +- a class path in `module:ClassName` format + +For a plugin, key its settings by the exact class path: + +```yaml +policy: + generation: + refit_transport: "my_pkg.refit:MyCheckpointEngine" + refit_cfg: + "my_pkg.refit:MyCheckpointEngine": + update_weights_bucket_memory_ratio: 0.05 + transport: custom +``` + +`update_weights_bucket_memory_ratio` is the fraction of fixed total GPU memory +used by each transfer bucket. Its Pydantic default is `0.05`. The +synchronizer queries every policy and rollout worker, uses the smallest reported +GPU capacity, and computes +`minimum_total_memory_bytes * update_weights_bucket_memory_ratio`, rounded down +to a MiB. The resolved size is fixed for the synchronizer lifetime. NIXL owns +two transfer buffers, so its total allocation is twice the configured ratio. + +The factory passes the resolved `bucket_size` in bytes plus the selected backend +kwargs to the backend constructor. + +## Backend Interface + +Backends subclass `nemo_rl.utils.checkpoint_engines.base.CheckpointEngine`. + +```python +from collections.abc import AsyncGenerator, Generator +from typing import Any + +import torch + +from nemo_rl.utils.checkpoint_engines.base import CheckpointEngine + + +class MyCheckpointEngine(CheckpointEngine): + def __init__(self, bucket_size: int, transport: str) -> None: + self.bucket_size = bucket_size + self.transport = transport + + def prepare(self) -> Any: + """Allocate or register buffers and return Ray-serializable metadata.""" + ... + + def get_target_weight_layout(self) -> dict[str, Any] | None: + """Return this policy rank's destination layout, if sharding weights.""" + ... + + def init_policy_process_group( + self, + *, + worker_rank: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[Any], + ) -> None: + """Connect a policy worker to its transfer peer.""" + ... + + def init_rollout_process_group( + self, + *, + rollout_rank: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[Any], + ) -> None: + """Connect a rollout worker to its transfer peer.""" + ... + + def finalize(self) -> None: + """Release per-refit state if the backend owns any.""" + ... + + async def send_weights( + self, + weights: Generator[tuple[str, torch.Tensor], None, None], + ) -> None: + """Send `(name, tensor)` weights from the policy side.""" + ... + + async def receive_weight_batches( + self, + ) -> AsyncGenerator[list[tuple[str, torch.Tensor]], None]: + """Yield `(name, tensor)` batches on the generation side.""" + ... +``` + +The `weights` generator is consumed once. `receive_weight_batches()` should +yield tensors with original parameter names and values. vLLM loads each yielded +batch immediately. + +A backend that enables `shard_expert_weights` must implement +`get_target_weight_layout()`. It returns `None` on policy ranks without a +rollout peer; otherwise it returns the destination layout used to filter and +slice the policy iterator. + +The built-in NIXL backend accepts `release_after_refit`. When enabled, +`finalize()` deregisters and frees its transfer buffers. A subsequent +`prepare()` allocates and registers new buffers before returning metadata. The +agent remains live, and the default retains the buffers as well for lower +latency. + +## Worker Integration + +Concrete policy workers opt into `PolicyCheckpointEngineMixin` beside their +backend-specific send mixin. `AbstractPolicyWorker` does not expose +checkpoint-engine methods, so value workers and other subclasses do not inherit +unused RPCs. The synchronizer invokes `checkpoint_engine_rpc()` for each +lifecycle step: creating the backend, preparing metadata, joining the backend +topology, sending weights, and finalizing the backend. Each concrete policy +worker supplies the iterator used by `send_weights_via_checkpoint_engine()`: + +- Megatron streams `_iter_params_with_optional_kv_scales()`. +- DTensor/FSDP2 streams the same local DTensor conversion path used by IPC and + NCCL refit. + +Some policy iterators materialize weights through distributed collectives. A +checkpoint backend must still drain the iterator on policy ranks without a +rollout peer so those collectives are entered by every required rank. + +vLLM generation workers forward checkpoint-engine calls through +`collective_rpc()` into vLLM internal workers. A normal vLLM worker uses +`VllmInternalWorkerExtension`, which contains the generic full and FP8 loaders +but no checkpoint-engine lifecycle methods. Enabling checkpoint-engine refit +selects `VllmInternalWorkerExtensionWithCheckpointEngine`, which adds backend +creation, metadata preparation, receiving, and sharded-expert dispatch. Its +explicit full-weight path delegates complete HF tensors to +`model.load_weights()` when `shard_expert_weights` is false. With +sharded-expert refit, it instead loads supported local expert shards through +validated destination-local views of canonical vLLM expert parameters. Dense +or otherwise unhandled tensors use the full-weight path. Before advertising a +sharded layout, the worker checks the physical expert storage shape and +backend. The vLLM worker prints timing for each update: + +```text +[vLLM refit] Loaded ... via checkpoint engine; bytes=... total=... receive=... load=... +``` + +NeMo RL pins the tested vLLM version, but the sharded MoE path still depends on +vLLM's canonical expert parameter layout. A version bump fails setup if the +storage dimensions or backend change, and the vLLM unit test compares batched +W1, W3, and W2 destination-local copies against vLLM's normal full-weight TP +loading. The residual silent-error risk is a same-shape layout semantic change. +A vLLM bump must therefore run `tools/refit_verifier.py`; use +`shard_expert_weights: false` until that verification passes. + +Async vLLM uses `checkpoint_engine_rpc_async()` and resolves nested +`collective_rpc()` awaitables, futures, and Ray object refs before reporting +success. + +## NIXL Backend + +The built-in NIXL backend is selected with `refit_transport: nixl`. It currently uses: + +- NIXL agents for memory registration and transfer +- ZMQ control messages for bucket metadata and completion notifications +- two reusable transfer buffers per worker +- staged bucket copies from policy tensors into NIXL buffers +- `split_weight_chunks()` and `merge_weight_chunk_batches()` for tensors larger + than one bucket + +The current topology is paired policy-to-rollout transfer. Policy rank `i` +sends to rollout rank `i` when `i < rollout_world_size`; extra policy workers do +not send. A rollout worker connects to the policy metadata entry at its rollout +rank, so production runs should allocate at least as many policy workers as +rollout workers for this backend. + +When sharded-expert refit is enabled, rollout metadata also contains the actual +vLLM destination layout for that worker. Each expert parameter reports whether +vLLM uses expert placement, its local global-expert IDs, and any remaining TP +coordinate. The layout also includes the missing-layer prefixes that vLLM uses +for pipeline-parallel loading. The paired policy worker drops weights absent from +the destination stage, then slices experts for TP or filters complete experts +for EP before filling NIXL buckets. This avoids deriving vLLM ownership from +Ray/global rank ordering. The destination metadata is authoritative; the NIXL +backend does not accept a source-side target-TP hint. + +`device` controls the staged transfer-buffer device: + +- `cuda`: allocate CUDA buffers and use CUDA-capable NIXL/UCX transfer. If + CuPy is available, CUDA buffers are allocated through CuPy before being + wrapped as torch tensors. +- `cpu`: allocate host buffers, pinned when CUDA is available. + +`backend_name` defaults to `UCX`. `device_list` restricts the local UCX network +devices and is independent of the distributed world size. The same list remains +valid when adding or removing homogeneous nodes; update it only when the +per-node HCA names or topology change. Omitting `device_list` lets UCX discover +available devices, but the NIXL 1.3 runtime used for validation defaults +`MAX_RMA_RAILS` to `2`, so that portable configuration does not reproduce the +validated eight-rail performance. For tuned runs, use devices available on +every participating node and keep `MAX_RMA_RAILS` no larger than the number of +usable selected rails. Values in `backend_init_params` are converted to strings +before creating the NIXL backend. + +Prefer `backend_init_params.device_list` over `UCX_NET_DEVICES` for per-run +selection because it is recorded with the run configuration. Both constrain +UCX discovery rather than overriding one another, so conflicting values can +exclude the intended devices. Reserve `UCX_NET_DEVICES` for a cluster-wide +override and normally configure only one of the two. + +The validated cluster omits `mlx5_3` because it maps to the Ethernet-link-layer +interface `enp90s0np0` on the `10.65.x.x/31` network, while the eight selected +HCAs use the InfiniBand link layer and map to `ibp*` interfaces on the +`100.126.x.x/16` RDMA data fabric. This mapping is cluster-specific; use +`ibdev2netdev` and inspect each RDMA port's `link_layer` instead of assuming +that device index 3 should always be excluded. + +### Validated Full-Model Layout + +A DeepSeek-V3 BF16 run validated different source and destination layouts: +Megatron TP1/PP16/EP16 across 256 policy workers refit vLLM TP32/PP1/EP1 across +32 rollout workers. The destination-reported layout drove PP filtering and TP +expert slicing without requiring the policy and rollout rank layouts to match. +Each rollout rank received 45,395 destination-local tensors in 18 batches, or +69.95 GiB. That cluster was tuned with eight explicitly selected HCAs and +`MAX_RMA_RAILS=8`; those device names are not portable defaults. Performance +and correctness-control results are recorded in the +[user guide](../guides/checkpoint-engine-refit.md#deepseek-v3-benchmark). + +## NIXL Preinit + +NIXL/UCX backend creation can be expensive if it first happens in the critical +path. The current code preinitializes NIXL agents in two places when the config +selects `refit_transport: nixl`: + +- policy worker construction +- vLLM internal worker construction, via vLLM's `worker_cls` hook + +NeMo RL passes the normalized `refit_cfg.nixl` settings through +`VllmConfig.additional_config`. `NixlVllmWorker` creates and retains the +preinit agent before calling vLLM's worker constructor. The preinit path uses +the configured `backend_name` and `backend_init_params`; logs usually show +NIXL agents named `preinit-...` during worker setup. + +`worker_cls` remains the early-construction hook for NIXL preinitialization. +`worker_extension_cls` is selected separately: the base extension is used +without checkpoint-engine refit, and the checkpoint-engine subclass is used +when the feature is enabled. + +## How NIXL Supports Fault Tolerance + +NIXL is the transfer layer. It does not create, remove, or replace Ray/vLLM +actors, and it does not route rollout requests. + +In NeMo RL, NIXL supports fault tolerance in three concrete ways: + +- **Transport errors become refit errors.** With UCX peer error handling + enabled, a lost peer can be reported to NIXL instead of leaving the transfer + waiting indefinitely. +- **Failed refits are propagated.** The NIXL backend raises when a read cannot + start or when `check_xfer_state()` reports `ERR`; vLLM reports the failed + weight update, and `CheckpointEngineWeightSynchronizer` raises for the refit. +- **A restarted synchronizer can use fresh peers.** `shutdown()` disconnects + the current peers. With `release_after_refit: true`, it also deregisters and + frees transfer buffers. The next `init_communicator()` registers new buffers, + exchanges `prepare()` metadata, and installs a new policy-to-rollout mapping. + +So the recovery model is fail the current refit, change the rollout actor set +outside NIXL, rebuild the checkpoint-engine communicator, then run a full refit +with fresh metadata before routing prompts to the new set. + +`tools/nixl_elastic_rollout_demo.py` demonstrates this teardown, metadata +exchange, and reinitialization sequence with synthetic weights. Actor creation, +removal, health checks, and request routing remain orchestration concerns. + +## Adding Another Backend + +To add a backend: + +1. Implement a `CheckpointEngine` subclass. +2. Accept `bucket_size` in bytes in the constructor. +3. Return only Ray-serializable metadata from `prepare()`. +4. Implement policy and rollout peer setup using the combined metadata list. +5. Stream policy weights from the input generator without replaying it. +6. Yield vLLM-loadable `(name, tensor)` batches from `receive_weight_batches()`. +7. Add backend-specific config under `refit_cfg.`. +8. Use a `module:ClassName` `refit_transport` value, or add a short-name + mapping in `create_checkpoint_engine()` if the backend should be built in. +9. Run a non-colocated GRPO job and verify the `[vLLM refit]` timing line. + +Current limitations: + +- Checkpoint-engine refit targets non-colocated policy-to-vLLM refit. +- SGLang and Megatron generation do not implement checkpoint-engine refit; + [issue #3288](https://github.com/NVIDIA-NeMo/RL/issues/3288) tracks + generation-side support. Megatron and DTensor policy backends are supported + when the generation backend is vLLM. +- The built-in NIXL backend uses paired policy-to-rollout transfer only. +- Sharded vLLM EP refit supports static expert ownership and canonical + unquantized Triton expert storage. Dynamic EPLB, redundant experts, and + quantized or shuffled layouts require a destination layout adapter and are + rejected during setup. diff --git a/docs/design-docs/sparse-delta-refit.md b/docs/design-docs/sparse-delta-refit.md index 830d7f1bcec..3c56d35cc57 100644 --- a/docs/design-docs/sparse-delta-refit.md +++ b/docs/design-docs/sparse-delta-refit.md @@ -226,8 +226,9 @@ Mamba, padded or tied weights, grouped exports, adapters, and custom Bridge postprocessing all follow Bridge's canonical export semantics. Baselines use file-backed `torch.from_file` tensors by default; -`refit_cfg.baseline.in_memory: true` keeps them in RAM. File backing reduces -anonymous resident-memory pressure but does not reduce logical baseline bytes. +`refit_cfg.sparse.baseline.in_memory: true` keeps them in RAM. File backing +reduces anonymous resident-memory pressure but does not reduce logical baseline +bytes. Baseline initialization also returns each canonical tensor's name, shape, and dtype. The synchronizer merges that metadata and asks every vLLM worker to @@ -464,14 +465,15 @@ policy: backend: vllm refit_transport: vllm_s3_sparse # or vllm_zmq_sparse refit_cfg: - delta_compression: - encoding: xor # overwrite is selected automatically for opaque loaders - storage: - s3_bucket: my-refit-bucket # required only for vllm_s3_sparse - s3_region: us-east-1 - baseline: - in_memory: false - verify_samples_per_payload: 0 + sparse: + delta_compression: + encoding: xor # overwrite is selected automatically for opaque loaders + storage: + s3_bucket: my-refit-bucket # required only for vllm_s3_sparse + s3_region: us-east-1 + baseline: + in_memory: false + verify_samples_per_payload: 0 colocated: enabled: false vllm_cfg: @@ -483,9 +485,10 @@ policy: zmq_refit_server_port: null ``` -`refit_cfg` is optional; its Pydantic models resolve and log all defaults. S3 -fails during setup unless `refit_cfg.storage.s3_bucket` is nonempty. Its region -and key prefix default to `us-east-1` and `nemo-rl-refit`. ZeroMQ requires +`refit_cfg.sparse` is optional; its Pydantic model resolves and logs all +defaults. S3 fails during setup unless +`refit_cfg.sparse.storage.s3_bucket` is nonempty. Its region and key prefix +default to `us-east-1` and `nemo-rl-refit`. ZeroMQ requires routable TCP access to the relay port. The HTTP and ZeroMQ servers are plaintext, so use a trusted or encrypted network. When `http_refit_api_key_env_var` is set, the named variable must contain the same @@ -494,11 +497,11 @@ interfaces without a key emits a warning. | Control | Default | |---|---:| -| `refit_cfg.delta_compression.encoding` | `xor` | -| `refit_cfg.storage.s3_bucket` | none; required for S3 | -| `refit_cfg.storage.s3_region` | `us-east-1` | -| `refit_cfg.baseline.in_memory` | `false` | -| `refit_cfg.verify_samples_per_payload` | `0` | +| `refit_cfg.sparse.delta_compression.encoding` | `xor` | +| `refit_cfg.sparse.storage.s3_bucket` | none; required for S3 | +| `refit_cfg.sparse.storage.s3_region` | `us-east-1` | +| `refit_cfg.sparse.baseline.in_memory` | `false` | +| `refit_cfg.sparse.verify_samples_per_payload` | `0` | Export chunks are capped by `sparse_bucket_size_bytes` and the packed tensor limit. The S3 defaults were selected by balanced 120B sweeps. Increase one @@ -518,7 +521,7 @@ logged config: ```bash uv run python examples/run_grpo.py \ --config examples/configs/recipes/llm/grpo-qwen3-30ba3b-4n8g-megatron-zmq-deltaweight-noncolocated.yaml \ - policy.generation.refit_cfg.verify_samples_per_payload=32 + policy.generation.refit_cfg.sparse.verify_samples_per_payload=32 ``` ## Metrics and profiling diff --git a/docs/guides/checkpoint-engine-refit.md b/docs/guides/checkpoint-engine-refit.md new file mode 100644 index 00000000000..d8a8d39e929 --- /dev/null +++ b/docs/guides/checkpoint-engine-refit.md @@ -0,0 +1,290 @@ +# Checkpoint-Engine Refit + +Checkpoint-engine refit updates non-colocated generation workers directly from +policy workers. The built-in backend is NIXL, which can use UCX/RDMA for large +policy-to-vLLM refits. + +Use it only for non-colocated vLLM generation: + +- `policy.generation.backend=vllm` +- `policy.generation.colocated.enabled=false` +- `policy.generation.refit_transport=nixl` + +Colocated generation still uses IPC/HTTP refit. Non-colocated generation without +checkpoint-engine refit still uses the NCCL collective update path. + +`examples/configs/grpo_math_8B_megatron_nixl.yaml` is a complete two-node +example built as an overlay on the standard 8B Megatron recipe. + +For a minimal run, start from `examples/configs/grpo_math_1B.yaml`, set +`policy.generation.colocated.enabled=false`, and set +`policy.generation.refit_transport=nixl`. The base config exposes the NIXL +defaults under `refit_cfg.nixl` so individual settings can be overridden. + +## Enable NIXL + +Select NIXL and configure its scoped refit settings: + +```yaml +policy: + generation: + backend: vllm + colocated: + enabled: false + resources: + num_nodes: 1 + gpus_per_node: 8 + refit_transport: nixl + refit_cfg: + nixl: + update_weights_bucket_memory_ratio: 0.05 + device: cuda + release_after_refit: false + backend_name: UCX + backend_init_params: + engine_config: MAX_RMA_RAILS=8 + device_list: "mlx5_0,mlx5_1,mlx5_2,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_8" +``` + +Key settings: + +| Key | Meaning | +|---|---| +| `update_weights_bucket_memory_ratio` | Fraction of fixed total GPU memory used by each transfer buffer. Defaults to `0.05`. NIXL allocates two buffers, so the default reserves 10% in total. | +| `device` | `cuda` uses GPU RDMA buffers. `cpu` uses host-pinned buffers and is mainly a fallback. | +| `release_after_refit` | When `true`, deregister and free transfer buffers after every refit. The next refit allocates and registers them again. Default `false` retains them for throughput. | +| `shard_expert_weights` | Sends destination-local MoE expert shards for TP/EP and omits weights absent from each vLLM PP stage. | +| `backend_init_params` | NIXL backend parameters such as UCX peer error handling, device lists, and UCX engine config. | + +The built-in NIXL topology is paired policy-to-rollout transfer, so allocate at +least as many policy workers as rollout workers. + +The driver queries fixed total GPU memory from every policy and vLLM worker +before creating the engines and uses the smallest capacity. It multiplies that +capacity by `update_weights_bucket_memory_ratio` and rounds down to a MiB. For +example, the default `0.05` selects a 4 GiB bucket on an 80 GiB GPU. Because +NIXL uses two buffers, that configuration reserves about 8 GiB per participating +engine. The selected size is fixed for the synchronizer lifetime. + +## Runtime Setup + +NIXL must be importable in every participating environment: + +- policy worker environment +- vLLM worker environment, including async vLLM workers + +Keep refit selection and NIC/rail selection in YAML or Hydra overrides so they +are captured in the logged run configuration. Prefer +`refit_cfg.nixl.backend_init_params.device_list` for a validated topology. +Cluster-wide `UCX_*` environment variables remain useful for UCX behavior that +NIXL does not expose through `backend_init_params`: + +```sh +export UCX_TLS=rc,cuda_copy,cuda_ipc,self,sm +export UCX_IB_ROCE_REACHABILITY_MODE=all +export UCX_MEMTYPE_CACHE=n +export UCX_MAX_RNDV_RAILS=8 +export UCX_WARN_UNUSED_ENV_VARS=n +export NIXL_LOG_LEVEL=INFO +``` + +Do not normally set both `device_list` and `UCX_NET_DEVICES`: they are two +filters on UCX device discovery, and a disagreement can remove the intended +rails. Use `UCX_NET_DEVICES` only as a cluster-wide override when the run config +cannot carry `device_list`. + +When vLLM starts nested Ray workers, copy the transport variables into those +workers: + +```sh +export VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=MELLANOX_ +export VLLM_RAY_EXTRA_ENV_VARS_TO_COPY=LD_LIBRARY_PATH,NIXL_LOG_LEVEL,NVIDIA_VISIBLE_DEVICES,UCX_NET_DEVICES,UCX_TLS,UCX_IB_ROCE_REACHABILITY_MODE,UCX_MEMTYPE_CACHE,UCX_MAX_RNDV_RAILS,UCX_WARN_UNUSED_ENV_VARS +``` + +Use NIC names that exist on the target nodes. With `UCX_LOG_LEVEL=info`, UCX +should report an RDMA transport such as `rc_mlx5`; TCP-only transport will be +much slower. + +## Performance Notes + +Use `device: cuda` for the fast path when NIXL/UCX can register CUDA memory. +For large MoE models with vLLM, leave enough GPU memory for NIXL buffers. The +following destination layout was validated with DeepSeek-V3 BF16: + +```yaml +policy: + generation: + vllm_cfg: + tensor_parallel_size: 32 + pipeline_parallel_size: 1 + expert_parallel_size: 1 + async_engine: true + gpu_memory_utilization: 0.82 + vllm_kwargs: + moe_backend: triton + refit_transport: nixl + refit_cfg: + nixl: + update_weights_bucket_memory_ratio: 0.05 + device: cuda + release_after_refit: false + shard_expert_weights: true +``` + +With `shard_expert_weights`, each vLLM worker reports its live destination layout +during checkpoint-engine setup. For pipeline parallelism, the source omits +every weight that vLLM marks as absent from the destination stage. For +tensor-parallel MoE, the source sends that TP rank's slice of every expert. For +expert-parallel MoE, the source sends complete experts only to ranks that own +them, followed by any intra-expert TP slice reported by vLLM. Any static +ownership map reported by vLLM is supported. The reported destination layout is +authoritative; do not configure a separate target-TP or rank-derived sharding +hint. + +Set `shard_expert_weights: false` to use the reference full-weight loader. This is +the fallback for a new vLLM layout or loader contract; fallback cannot happen +after transfer starts because the sender has already discarded non-local +shards. Sharded setup therefore validates vLLM's canonical parameter layout +and source shard shapes and fails before loading when those contracts do not +match. + +Dynamic expert load balancing and redundant experts are not supported because +ownership can change after metadata exchange. The direct-copy load path accepts +only unquantized Triton expert storage; FP8 and backends that transpose or +shuffle expert weights are rejected during setup. This includes FlashInfer +TRT-LLM MXFP8 layouts that reorder W13 to W31 and shuffle weights and scales. + +### FP8 Compatibility + +FP8 model weights and FP8 KV-cache scales are separate features: + +| Configuration | Status | +|---|---| +| NIXL with `shard_expert_weights: false` and FP8 vLLM weights | Supported through the existing full-weight `fp8.load_weights` path. | +| NIXL with `shard_expert_weights: true` and FP8 or MXFP8 vLLM weights | Unsupported. Setup or loading fails explicitly because destination-local expert loading does not yet implement quantized, transposed, or shuffled weight-and-scale layouts. | +| Megatron policy with FP8 KV-cache scales | Supported; the scales are appended to the checkpoint-engine weight stream and processed after loading. | +| DTensor or DTensor v2 policy with FP8 KV-cache scales | Unsupported; the policy worker raises `NotImplementedError`. | + +Sharded FP8 expert refit is a loader-layout limitation, not a NIXL transport +limitation. Supporting it requires a versioned destination-layout adapter for +the quantized weights, scales, and any vLLM post-load transpose or shuffle. + +A full-weight FP8 end-to-end validation on 2026-07-17 used a BF16 Megatron +Qwen2.5-0.5B policy, an FP8 vLLM rollout model, two eight-GPU nodes, asynchronous +rollout, CUDA-buffer NIXL, and a 256 MiB bucket. Two refits completed in 1.116 s +and 0.195 s. `tools/refit_verifier.py` reported mean/max absolute logprob +differences of 0.09474/0.18032 and an average probability multiplier of 1.10153. +These differences include BF16-to-FP8 quantization error, so this validates the +full-weight FP8 control and transfer path rather than bitwise transport parity. + +Only destination-local expert tensors use this direct-copy path. Dense and +unhandled tensors continue through vLLM's standard `load_weights` path. CUDA +NIXL still stages transfers through GPU RDMA buffers, but tensors spanning +multiple buffers are reassembled on CPU before that standard loader to limit +peak GPU memory. + +When vLLM EP is larger than TP, NeMo RL currently requires `async_engine: false`. +For non-MoE models, `shard_expert_weights` only applies pipeline-stage filtering; +dense tensors are not sharded. +Expert sharding requires HF expert names that map to vLLM `w13_weight` and +`w2_weight` parameters. + +Avoid `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for CUDA-buffer NIXL +refit unless that workload has been explicitly validated with it. + +Set `release_after_refit: true` when the two registered buffers must be returned +between refits. Finalization then deregisters both buffers, releases their +dedicated CuPy pool, and retains the NIXL agent and control endpoint. Before the +next refit, `prepare()` allocates and registers two new buffers. This saves +twice the resolved bucket size of resident GPU memory between refits at the cost +of allocation, registration, and metadata exchange on every refit. Reallocation +uses the size selected when the synchronizer was initialized. Leave it `false` +for the benchmark path below. + +### DeepSeek-V3 Benchmark + +The full-model benchmark used 36 eight-GPU nodes: 32 policy nodes +with Megatron TP1/PP16/EP16 and four rollout nodes with vLLM TP32/PP1/EP1. NIXL +used CUDA buffers, UCX over eight RDMA rails, a 4096 MiB bucket, Triton MoE, and +`shard_expert_weights: true`. Each vLLM rank loaded 45,395 destination-local tensors +in 18 batches, totaling 69.95 GiB. + +| Measurement | NIXL GPU RDMA | NCCL | NIXL improvement | +|---|---:|---:|---:| +| Dedicated async refit timer | 9.25 s | 14.21 s | 1.54x, 35% less time | +| Matched synchronous verifier | 10.92 s | 15.52 s | 1.42x, 30% less time | + +Compare results within one row only; async and synchronous vLLM have different +control-path overhead. + +A same-build synchronous regression run on 2026-07-15 changed only +`shard_expert_weights`: the full-expert path took 36.812 s and the sharded-expert +path took 10.923 s, a 3.37x speedup and 70% less time. The verifier's recorded +output IDs, vLLM logprobs, policy logprobs, per-row errors, and aggregate metrics +were identical between paths. Both paths reported mean/max policy-to-vLLM +logprob differences of 0.03227/0.09738; this comparison establishes parity with +the full-weight loader, not absolute equivalence between the two model stacks. + +## Fault Tolerance Boundary + +NIXL is the transfer layer. It does not add, remove, or replace Ray/vLLM +rollout actors, and it does not route rollout requests. + +What NIXL provides: + +- UCX peer error handling can turn a lost peer into a transport error instead + of an indefinite wait. +- The NIXL backend raises when a read cannot start or completes with `ERR`. +- `CheckpointEngineWeightSynchronizer` propagates a failed update as a failed + refit. +- Reinitializing the synchronizer exchanges fresh NIXL metadata for the current + policy and rollout actor set. + +Enable UCX peer error handling when failed peers should surface promptly: + +```yaml +policy: + generation: + refit_transport: nixl + refit_cfg: + nixl: + backend_init_params: + ucx_error_handling_mode: peer +``` + +Changing the rollout actor set is orchestration outside NIXL: stop routing to +the old actor, create or remove the Ray/vLLM actor, shut down and reinitialize +the checkpoint-engine communicator, then run a full refit before routing prompts +to the new set. + +`tools/nixl_elastic_rollout_demo.py` exercises that communicator teardown and +reinitialization sequence with synthetic weights. It demonstrates the NIXL +lifecycle boundary, not automatic Ray actor recovery. + +## Verify + +The driver log should show: + +```text +Using checkpoint-engine refit backend: nixl +``` + +Each vLLM update should also print: + +```text +[vLLM refit] Loaded ... via checkpoint engine; bytes=... total=... receive=... load=... +``` + +Use `tools/refit_verifier.py` for a refit correctness smoke test: + +```sh +uv run --extra mcore --extra vllm python tools/refit_verifier.py \ + --model_name /path/to/model \ + --tp_size 1 \ + --ep_size 1 \ + --pp_size 1 +``` + +That verifier compares vLLM and Megatron logprobs after a refit. It is useful +for model/refit correctness, while the NIXL transport path is confirmed by the +non-colocated GRPO log markers above. diff --git a/docs/guides/refit.md b/docs/guides/refit.md new file mode 100644 index 00000000000..442fe36f191 --- /dev/null +++ b/docs/guides/refit.md @@ -0,0 +1,100 @@ +# Weight Refit: Choosing a Transport + +Weight refit copies updated policy weights into the rollout model. Choose the +topology first, then select one non-colocated transport with +`policy.generation.refit_transport`. + +## Pick a Transport + +| Topology | `refit_transport` | Transport | Use when | +|---|---|---|---| +| `colocated.enabled: true` | `null` | CUDA IPC or HTTP | Policy and rollout workers share GPUs; vLLM uses IPC and SGLang uses HTTP. | +| `colocated.enabled: false` | `null` | NCCL broadcast | You want the default full-weight path without extra dependencies. | +| `colocated.enabled: false` | `vllm_zmq_sparse` | Sparse delta over ZeroMQ | The link is bandwidth-limited and workers can reach a relay over TCP. | +| `colocated.enabled: false` | `vllm_s3_sparse` | Sparse delta through S3 | Workers communicate through shared object storage. | +| `colocated.enabled: false` | `nixl` | NIXL checkpoint engine | The cluster has a fast UCX/RDMA fabric for full-weight refit. | + +`null` is the default. The sparse transports read only `refit_cfg.sparse`; NIXL +reads only `refit_cfg.nixl`. Because one selector chooses the transport, sparse +delta and NIXL cannot both be active. + +## Constraints + +| Transport | Generation backend | Policy backend | Quantization and MoE | +|---|---|---|---| +| Colocated IPC/HTTP | vLLM or SGLang | DTensor or Megatron | Uses the generation backend's standard loader. | +| NCCL | vLLM or Megatron | DTensor or Megatron | Uses the standard full-weight loader. | +| Sparse delta | vLLM | Megatron | BF16/FP16, unquantized rollout only. | +| NIXL, full weights | vLLM | DTensor or Megatron | Supports the standard full-weight FP8 loader. DTensor FP8 KV-cache scale transfer is not yet supported. | +| NIXL, sharded experts | vLLM | DTensor or Megatron | Unquantized BF16/FP16 Triton MoE only; FP8/MXFP8 and dynamic expert placement are rejected. | + +Non-colocated SGLang generation is not supported. The NIXL restrictions are on +the generation backend; both Megatron and DTensor policy workers can send +weights. Sparse delta is currently limited to GRPO. NIXL is initialized by the +GRPO and distillation setup paths; PPO currently requires colocated generation. + +## Minimal Configuration + +Colocated refit needs no transport configuration: + +```yaml +policy: + generation: + colocated: + enabled: true + refit_transport: null +``` + +For non-colocated NCCL, change the topology and leave the selector unset: + +```yaml +policy: + generation: + colocated: + enabled: false + refit_transport: null +``` + +For sparse delta, select one data plane and configure its scope: + +```yaml +policy: + generation: + colocated: + enabled: false + refit_transport: vllm_zmq_sparse # or vllm_s3_sparse + refit_cfg: + sparse: + delta_compression: + encoding: xor + storage: + s3_bucket: null # required for vllm_s3_sparse +``` + +For NIXL, select the checkpoint engine and configure its scope: + +```yaml +policy: + generation: + colocated: + enabled: false + refit_transport: nixl + refit_cfg: + nixl: + update_weights_bucket_memory_ratio: 0.05 + device: cuda + backend_name: UCX + release_after_refit: false + shard_expert_weights: false +``` + +## Learn More + +- [Sparse Delta Refit](../design-docs/sparse-delta-refit.md) explains baseline, + compression, ZeroMQ, and S3 behavior. +- [Checkpoint-Engine Refit](checkpoint-engine-refit.md) covers NIXL setup, + performance tuning, FP8, sharded experts, and fault tolerance. +- [Checkpoint Engines](../design-docs/checkpoint-engines.md) describes the + checkpoint-engine protocol and implementation. +- [Training and Generation Backends](../about/backends.md) summarizes backend + compatibility. diff --git a/docs/index.md b/docs/index.md index aa3f85e76ce..1a140193b02 100644 --- a/docs/index.md +++ b/docs/index.md @@ -163,6 +163,20 @@ Extend a model's context window with YaRN RoPE scaling on the Megatron backend f Off-policy distillation across mismatched tokenizers — build a (student, teacher) projection matrix and run x-token KD via CUDA-IPC teacher logits. ::: +:::{grid-item-card} {octicon}`arrow-both` Weight Refit +:link: guides/refit +:link-type: doc + +Choose among colocated IPC, NCCL, sparse delta, and NIXL refit transports. +::: + +:::{grid-item-card} {octicon}`sync` Checkpoint-Engine Refit +:link: guides/checkpoint-engine-refit +:link-type: doc + +Use NIXL checkpoint-engine refit to update non-colocated vLLM generation workers from policy workers. +::: + :::: ## Advanced Topics @@ -288,6 +302,8 @@ guides/quantization-aware-rl.md guides/eagle3-speculative-decoding.md guides/yarn-long-context.md guides/xtoken-off-policy-distillation.md +guides/refit.md +guides/checkpoint-engine-refit.md guides/router-replay.md guides/muon-optimizer.md guides/dtensor-tp-accuracy.md @@ -323,6 +339,7 @@ design-docs/dependency-management.md design-docs/chat-datasets.md design-docs/generation.md design-docs/sparse-delta-refit.md +design-docs/checkpoint-engines.md design-docs/checkpointing.md design-docs/loss-functions.md design-docs/fsdp2-parallel-plan.md diff --git a/examples/configs/grpo_math_1B.yaml b/examples/configs/grpo_math_1B.yaml index 44ae519a3f9..88a946c61b0 100644 --- a/examples/configs/grpo_math_1B.yaml +++ b/examples/configs/grpo_math_1B.yaml @@ -339,8 +339,17 @@ policy: top_k: null stop_token_ids: null stop_strings: null - refit_transport: null # Set to "vllm_s3_sparse" or "vllm_zmq_sparse" for remote sparse-delta refit. - refit_cfg: null # Optional tuning and storage settings for remote sparse-delta refit. + # null = topology default (IPC colocated, NCCL non-colocated). + # Non-colocated vLLM also supports vllm_s3_sparse, vllm_zmq_sparse, and nixl. + refit_transport: null + refit_cfg: + nixl: + update_weights_bucket_memory_ratio: 0.05 + device: cuda + backend_name: UCX + backend_init_params: null + release_after_refit: false + shard_expert_weights: false mcore_generation_config: async_engine: false max_model_len: ${policy.max_total_sequence_length} # Engine-side max sequence length. diff --git a/examples/configs/grpo_math_8B_megatron_nixl.yaml b/examples/configs/grpo_math_8B_megatron_nixl.yaml new file mode 100644 index 00000000000..4fa0924c2dc --- /dev/null +++ b/examples/configs/grpo_math_8B_megatron_nixl.yaml @@ -0,0 +1,24 @@ +# GRPO with non-colocated NIXL/UCX checkpoint-engine refit +defaults: "grpo_math_8B_megatron.yaml" + +policy: + generation: + colocated: + enabled: false + resources: + gpus_per_node: 8 + num_nodes: 1 + refit_transport: nixl + refit_cfg: + nixl: + # fraction of total GPU memory used by each weight-transfer bucket + update_weights_bucket_memory_ratio: 0.05 + device: cuda + # Free registered transfer buffers between refits. Keep false for throughput. + release_after_refit: false + shard_expert_weights: false + backend_name: UCX + backend_init_params: null + +cluster: + num_nodes: 2 diff --git a/examples/configs/recipes/llm/grpo-qwen3-30ba3b-4n8g-megatron-zmq-deltaweight-noncolocated.yaml b/examples/configs/recipes/llm/grpo-qwen3-30ba3b-4n8g-megatron-zmq-deltaweight-noncolocated.yaml index 18ff88ca112..9367fd0fe49 100644 --- a/examples/configs/recipes/llm/grpo-qwen3-30ba3b-4n8g-megatron-zmq-deltaweight-noncolocated.yaml +++ b/examples/configs/recipes/llm/grpo-qwen3-30ba3b-4n8g-megatron-zmq-deltaweight-noncolocated.yaml @@ -23,11 +23,12 @@ policy: generation: refit_transport: vllm_zmq_sparse refit_cfg: - delta_compression: - encoding: xor - verify_samples_per_payload: 0 - baseline: - in_memory: false + sparse: + delta_compression: + encoding: xor + verify_samples_per_payload: 0 + baseline: + in_memory: false colocated: enabled: false resources: diff --git a/nemo_rl/algorithms/distillation.py b/nemo_rl/algorithms/distillation.py index 7818a8dd418..6f536ed616e 100644 --- a/nemo_rl/algorithms/distillation.py +++ b/nemo_rl/algorithms/distillation.py @@ -69,6 +69,10 @@ GenerationInterface, ) from nemo_rl.models.generation.vllm import VllmConfig, VllmGeneration +from nemo_rl.models.generation.vllm.config import ( + VLLM_SPARSE_REFIT_TRANSPORTS, + normalize_vllm_refit_config, +) from nemo_rl.models.policy import PolicyConfig from nemo_rl.models.policy.interfaces import ColocatablePolicyInterface from nemo_rl.models.policy.lm_policy import Policy @@ -81,6 +85,10 @@ from nemo_rl.utils.nsys import maybe_gpu_profile_step from nemo_rl.utils.timer import TimeoutChecker, Timer from nemo_rl.utils.venvs import make_actor_runtime_env +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) +from nemo_rl.weight_sync.factory import create_weight_synchronizer # =============================================================================== # Configuration @@ -217,14 +225,18 @@ def setup( assert generation_config is not None, ( "A generation config in the PolicyConfig is required for distillation" ) - if ( - generation_config["backend"] == "vllm" - and cast(VllmConfig, generation_config).get("refit_transport") is not None - ): - raise ValueError( - "Remote sparse refit is currently supported only by GRPO; distillation " - "support is tracked in https://github.com/NVIDIA-NeMo/RL/issues/3275." - ) + checkpoint_engine_config = None + if generation_config["backend"] == "vllm": + vllm_config = cast(VllmConfig, generation_config) + normalize_vllm_refit_config(vllm_config) + refit_transport = vllm_config.get("refit_transport") + if refit_transport in VLLM_SPARSE_REFIT_TRANSPORTS: + raise ValueError( + "Remote sparse refit is currently supported only by GRPO; " + "distillation support is tracked in " + "https://github.com/NVIDIA-NeMo/RL/issues/3275." + ) + checkpoint_engine_config = checkpoint_engine_refit_config(vllm_config) # Disallow SP + packing for dtensor path for cfg, who in ((policy_config, "student"), (teacher_config, "teacher")): @@ -594,12 +606,23 @@ def init_nemo_gym(): init_reference_model=False, ) - if student_generation is not None: + if checkpoint_engine_config is not None: + assert isinstance(student_generation, VllmGeneration) + student_generation.weight_synchronizer = create_weight_synchronizer( + policy=student_policy, + generation=student_generation, + generation_backend=backend, + colocated=colocated_inference, + train_cluster=train_cluster, + inference_cluster=inference_cluster, + ) + student_generation.weight_synchronizer.init_communicator() + elif student_generation is not None: state_dict_info = student_policy.prepare_refit_info() student_generation.prepare_refit_info(state_dict_info) # if it is not colocated inference, initialize collective communication for update weights - if not colocated_inference: + if not colocated_inference and checkpoint_engine_config is None: ip, port = train_cluster.get_master_address_and_port() print(f"Using ip: {ip}, port: {port} for collective communication", flush=True) train_world_size = train_cluster.world_size() diff --git a/nemo_rl/algorithms/grpo.py b/nemo_rl/algorithms/grpo.py index 2e3a472809f..6eb6c4f1592 100644 --- a/nemo_rl/algorithms/grpo.py +++ b/nemo_rl/algorithms/grpo.py @@ -101,7 +101,10 @@ from nemo_rl.models.generation.sglang.config import SGLangConfig from nemo_rl.models.generation.sglang.sglang_generation import SGLangGeneration from nemo_rl.models.generation.vllm import VllmConfig, VllmGeneration -from nemo_rl.models.generation.vllm.config import normalize_vllm_refit_config +from nemo_rl.models.generation.vllm.config import ( + VLLM_SPARSE_REFIT_TRANSPORTS, + normalize_vllm_refit_config, +) from nemo_rl.models.megatron.router_replay import ( configure_vllm_for_router_replay, router_replay_enabled, @@ -119,6 +122,10 @@ from nemo_rl.utils.nsys import maybe_gpu_profile_step from nemo_rl.utils.timer import TimeoutChecker, Timer from nemo_rl.utils.venvs import create_local_venv_on_each_node +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) +from nemo_rl.weight_sync.factory import create_weight_synchronizer # =============================================================================== # Configuration @@ -913,6 +920,7 @@ def _spinup_nemo_gym(base_urls, model_name): remote_transport = None remote_synchronizer_cls = None remote_baseline_init_refs: list[Any] = [] + checkpoint_engine_config = None # Dictionary to store worker initialization timing stats for logging worker_init_timing_metrics = {} @@ -1106,7 +1114,8 @@ def initialize_generation_with_policy( elif backend == "vllm": # vLLM generation: setup config, then initialize with policy generation_config = cast(VllmConfig, generation_config) - if generation_config.get("refit_transport") is not None: + refit_transport = generation_config.get("refit_transport") + if refit_transport in VLLM_SPARSE_REFIT_TRANSPORTS: # Keep optional remote transport dependencies off the default path. from nemo_rl.weight_sync.vllm_remote_sparse_weight_synchronizer import ( VllmRemoteSparseWeightSynchronizer, @@ -1120,6 +1129,9 @@ def initialize_generation_with_policy( ) assert remote_transport is not None remote_synchronizer_cls = VllmRemoteSparseWeightSynchronizer + elif refit_transport is not None: + checkpoint_engine_config = checkpoint_engine_refit_config(generation_config) + assert checkpoint_engine_config is not None if generation_config["vllm_cfg"]["precision"] == "fp8": assert loss_config.use_importance_sampling_correction, ( @@ -1257,8 +1269,21 @@ def init_vllm_then_policy(): # print the node IP and GPU ID of the policy workers for debugging policy.print_node_ip_and_gpu_id() + if generation_config.get("refit_transport") is not None and backend != "vllm": + raise NotImplementedError( + "Non-default refit transports are only supported for the vLLM " + f"generation backend, but policy.generation.backend={backend!r}. " + "Set policy.generation.refit_transport=null. Support for other " + "generation backends is tracked in " + "https://github.com/NVIDIA-NeMo/RL/issues/3288." + ) + # if it is not colocated inference, initialize collective communication for update weights - if not colocated_inference and remote_transport is None: + if ( + not colocated_inference + and remote_transport is None + and checkpoint_engine_config is None + ): t0 = time.perf_counter() ip, port = train_cluster.get_master_address_and_port() print(f"Using ip: {ip}, port: {port} for collective communication", flush=True) @@ -1296,7 +1321,6 @@ def init_vllm_then_policy(): ) # type: ignore ray.get(futures_train + futures_inference) worker_init_timing_metrics["collective_init_time_s"] = time.perf_counter() - t0 - if remote_transport is not None: t0 = time.perf_counter() assert isinstance(policy_generation, VllmGeneration) @@ -1310,13 +1334,32 @@ def init_vllm_then_policy(): api_key_env_var=generation_config["vllm_cfg"].get( "http_refit_api_key_env_var" ), - request_timeout_s=refit_config.request_timeout_s, + request_timeout_s=refit_config.sparse.request_timeout_s, baseline_init_refs=remote_baseline_init_refs, ) policy_generation.weight_synchronizer.init_communicator() worker_init_timing_metrics[f"vllm_{remote_transport}_sparse_init_time_s"] = ( time.perf_counter() - t0 ) + elif checkpoint_engine_config is not None: + t0 = time.perf_counter() + assert isinstance(policy_generation, VllmGeneration) + policy_generation.weight_synchronizer = create_weight_synchronizer( + policy=policy, + generation=policy_generation, + generation_backend=backend, + colocated=colocated_inference, + train_cluster=train_cluster, + inference_cluster=inference_cluster, + ) + policy_generation.weight_synchronizer.init_communicator() + worker_init_timing_metrics["vllm_checkpoint_engine_init_time_s"] = ( + time.perf_counter() - t0 + ) + print( + f"Using checkpoint-engine refit backend: {checkpoint_engine_config['backend']}", + flush=True, + ) else: state_dict_info = policy.prepare_refit_info() if policy_generation is not None: diff --git a/nemo_rl/algorithms/ppo.py b/nemo_rl/algorithms/ppo.py index 5bef74b5cef..9971fba15ef 100644 --- a/nemo_rl/algorithms/ppo.py +++ b/nemo_rl/algorithms/ppo.py @@ -70,6 +70,10 @@ from nemo_rl.models.generation.sglang.config import SGLangConfig from nemo_rl.models.generation.sglang.sglang_generation import SGLangGeneration from nemo_rl.models.generation.vllm import VllmConfig, VllmGeneration +from nemo_rl.models.generation.vllm.config import ( + VLLM_SPARSE_REFIT_TRANSPORTS, + normalize_vllm_refit_config, +) from nemo_rl.models.policy import MegatronConfig, PolicyConfig from nemo_rl.models.policy.interfaces import ColocatablePolicyInterface from nemo_rl.models.policy.lm_policy import Policy @@ -228,14 +232,23 @@ def setup( assert generation_config is not None, ( "A generation config in the PolicyConfig is required for PPO" ) - if ( - generation_config["backend"] == "vllm" - and cast(VllmConfig, generation_config).get("refit_transport") is not None - ): - raise ValueError( - "Remote sparse refit is currently supported only by GRPO; PPO support " - "is tracked in https://github.com/NVIDIA-NeMo/RL/issues/3275." - ) + if generation_config["backend"] == "vllm": + vllm_config = cast(VllmConfig, generation_config) + normalize_vllm_refit_config(vllm_config) + refit_transport = vllm_config.get("refit_transport") + if refit_transport in VLLM_SPARSE_REFIT_TRANSPORTS: + raise ValueError( + "Remote sparse refit is currently supported only by GRPO; PPO " + "support is tracked in " + "https://github.com/NVIDIA-NeMo/RL/issues/3275." + ) + if refit_transport is not None: + raise ValueError( + "Checkpoint-engine refit requires non-colocated generation, but " + "PPO currently requires colocated generation. Non-colocated PPO " + "support is tracked in " + "https://github.com/NVIDIA-NeMo/RL/issues/3275." + ) if "megatron_cfg" in policy_config and policy_config["megatron_cfg"]["enabled"]: policy_megatron_config = cast(MegatronConfig, policy_config["megatron_cfg"]) diff --git a/nemo_rl/modelopt/models/generation/vllm_quant_backend.py b/nemo_rl/modelopt/models/generation/vllm_quant_backend.py index de22810dabe..4f436492321 100644 --- a/nemo_rl/modelopt/models/generation/vllm_quant_backend.py +++ b/nemo_rl/modelopt/models/generation/vllm_quant_backend.py @@ -27,6 +27,7 @@ MODELOPT_REAL_QUANT_ZMQ_TIMEOUT_MS, matches_quant_ignore_pattern, ) +from nemo_rl.models.generation.vllm.checkpoint_engine import VllmCheckpointEngineMixin from nemo_rl.models.generation.vllm.vllm_backend import ( IPCWeightManifestError, VllmInternalWorkerExtension, @@ -654,3 +655,9 @@ def get_quantizer_stats(self) -> dict: "with_amax": with_amax, "positive_amax": positive_amax, } + + +class VllmQuantInternalWorkerExtensionWithCheckpointEngine( + VllmCheckpointEngineMixin, VllmQuantInternalWorkerExtension +): + """ModelOpt worker extension with checkpoint-engine refit support.""" diff --git a/nemo_rl/modelopt/models/generation/vllm_quant_patch.py b/nemo_rl/modelopt/models/generation/vllm_quant_patch.py index de581727222..a79ecb214d2 100644 --- a/nemo_rl/modelopt/models/generation/vllm_quant_patch.py +++ b/nemo_rl/modelopt/models/generation/vllm_quant_patch.py @@ -22,9 +22,9 @@ from modelopt.torch.quantization.calib.max import MaxCalibrator from modelopt.torch.quantization.nn.modules.tensor_quantizer import TensorQuantizer from modelopt.torch.quantization.plugins.vllm import disable_compilation -from vllm.v1.worker.gpu_worker import Worker as BaseWorker from nemo_rl.modelopt.utils import resolve_quant_cfg +from nemo_rl.models.generation.vllm.vllm_backend import NixlVllmWorker @contextmanager @@ -100,7 +100,7 @@ def calibrate_loop(model: Any = None) -> None: module.disable() -class FakeQuantWorker(BaseWorker): +class FakeQuantWorker(NixlVllmWorker): @torch.inference_mode() def determine_available_memory(self) -> int: model = self.model_runner.model diff --git a/nemo_rl/modelopt/models/generation/vllm_quant_worker.py b/nemo_rl/modelopt/models/generation/vllm_quant_worker.py index 3f797ec4b7d..3b4e95d6df8 100644 --- a/nemo_rl/modelopt/models/generation/vllm_quant_worker.py +++ b/nemo_rl/modelopt/models/generation/vllm_quant_worker.py @@ -26,6 +26,9 @@ from nemo_rl.models.generation.vllm.vllm_worker_async import ( VllmAsyncGenerationWorkerImpl, ) +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) _EXTRA_ENV_VARS = ( "VLLM_QUANT_CFG", @@ -45,8 +48,11 @@ def _configure_quant_engine_kwargs( cfg: VllmConfig, llm_kwargs: dict[str, Any], ) -> None: + extension_name = "VllmQuantInternalWorkerExtension" + if checkpoint_engine_refit_config(cfg) is not None: + extension_name += "WithCheckpointEngine" llm_kwargs["worker_extension_cls"] = ( - "nemo_rl.modelopt.models.generation.vllm_quant_backend.VllmQuantInternalWorkerExtension" + "nemo_rl.modelopt.models.generation.vllm_quant_backend." + extension_name ) real_quant = bool(cfg.get("real_quant")) if real_quant: diff --git a/nemo_rl/models/generation/__init__.py b/nemo_rl/models/generation/__init__.py index cc471b1b27b..2ca7a539913 100644 --- a/nemo_rl/models/generation/__init__.py +++ b/nemo_rl/models/generation/__init__.py @@ -18,6 +18,7 @@ from nemo_rl.models.generation.interfaces import GenerationConfig from nemo_rl.models.generation.vllm import VllmConfig +from nemo_rl.models.generation.vllm.config import VLLM_SPARSE_REFIT_TRANSPORTS TokenizerType = PreTrainedTokenizerBase @@ -46,7 +47,9 @@ def configure_generation_config( config = cast(VllmConfig, config) # set load_format config["vllm_cfg"]["load_format"] = ( - "auto" if is_eval or config.get("refit_transport") else "dummy" + "auto" + if is_eval or config.get("refit_transport") in VLLM_SPARSE_REFIT_TRANSPORTS + else "dummy" ) speculative_config = config.get("vllm_kwargs", {}).get("speculative_config") if speculative_config and not is_eval and not has_refit_draft_weights: diff --git a/nemo_rl/models/generation/interfaces.py b/nemo_rl/models/generation/interfaces.py index 4584ee0b1e5..6a76cb40861 100644 --- a/nemo_rl/models/generation/interfaces.py +++ b/nemo_rl/models/generation/interfaces.py @@ -177,6 +177,17 @@ class ColocationConfig(TypedDict): resources: OptionalResourcesConfig +class CheckpointEngineConfig(TypedDict): + """Normalized internal configuration for checkpoint-engine refit.""" + + # "nixl" or a "module:ClassName" path to a CheckpointEngine implementation + backend: str + # fraction of total GPU memory used by each transfer bucket + update_weights_bucket_memory_ratio: float + # per-backend constructor kwargs, keyed by the configured backend string + engine_kwargs: dict[str, dict[str, Any]] + + class GenerationConfig(TypedDict): """Configuration for generation.""" diff --git a/nemo_rl/models/generation/vllm/checkpoint_engine.py b/nemo_rl/models/generation/vllm/checkpoint_engine.py new file mode 100644 index 00000000000..4a9a42bdba5 --- /dev/null +++ b/nemo_rl/models/generation/vllm/checkpoint_engine.py @@ -0,0 +1,202 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import time +from typing import TYPE_CHECKING, Any + +import torch + +from nemo_rl.models.generation.vllm.config import VllmConfig +from nemo_rl.models.generation.vllm.refit_loader import ( + VllmShardedExpertRefitMixin, +) +from nemo_rl.utils.nsys import wrap_with_nvtx_name +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) + +if TYPE_CHECKING: + from nemo_rl.utils.checkpoint_engines.base import CheckpointEngine + + +NIXL_VLLM_WORKER = "nemo_rl.models.generation.vllm.vllm_backend.NixlVllmWorker" +_NIXL_CONFIG_KEY = "nemo_rl_checkpoint_engine" + + +def configure_nixl_worker(config: VllmConfig, vllm_kwargs: dict[str, Any]) -> None: + """Configure vLLM's worker hook for early NIXL initialization.""" + checkpoint_config = checkpoint_engine_refit_config(config) + if checkpoint_config is None or checkpoint_config["backend"] != "nixl": + return + + worker_cls = vllm_kwargs.setdefault("worker_cls", NIXL_VLLM_WORKER) + if worker_cls != NIXL_VLLM_WORKER: + raise ValueError( + "NIXL checkpoint-engine refit requires vllm_kwargs.worker_cls to " + f"be unset or {NIXL_VLLM_WORKER}." + ) + + additional_config = dict(vllm_kwargs.get("additional_config") or {}) + additional_config[_NIXL_CONFIG_KEY] = checkpoint_config + vllm_kwargs["additional_config"] = additional_config + + +def preinit_nixl_from_vllm_config(vllm_config: Any) -> Any: + """Create the NIXL preinit agent carried by a vLLM internal worker.""" + checkpoint_config = vllm_config.additional_config.get(_NIXL_CONFIG_KEY) + if checkpoint_config is None: + return None + + from nemo_rl.utils.checkpoint_engines.nixl import ( + preinit_nixl_agent, + resolve_nixl_backend_kwargs, + ) + + backend_name, backend_init_params = resolve_nixl_backend_kwargs( + checkpoint_config["engine_kwargs"]["nixl"] + ) + return preinit_nixl_agent( + backend_name=backend_name, backend_init_params=backend_init_params + ) + + +def resolve_rollout_rank(rank_prefix: int, rollout_world_size: int) -> int: + rank = torch.distributed.get_rank() + if torch.distributed.get_world_size() == rollout_world_size: + # External DP ranks are already global; adding the prefix would double-count. + return rank + return rank_prefix + rank + + +class VllmCheckpointEngineMixin(VllmShardedExpertRefitMixin): + """Checkpoint-engine lifecycle for vLLM workers.""" + + checkpoint_engine: "CheckpointEngine" + + def checkpoint_engine_total_memory_bytes(self) -> int: + device = torch.cuda.current_device() + return torch.cuda.get_device_properties(device).total_memory + + def _load_hf_weights(self, policy_weights: list[tuple[str, torch.Tensor]]) -> None: + if self.checkpoint_engine.shard_expert_weights: + self._load_sharded_expert_weights(policy_weights) + return + super()._load_hf_weights(policy_weights) + + def init_checkpoint_engine( + self, backend: str, bucket_size_bytes: int, engine_kwargs: dict[str, Any] + ) -> None: # pragma: no cover + if getattr(self, "checkpoint_engine", None) is not None: + return + + from nemo_rl.utils.checkpoint_engines.base import create_checkpoint_engine + + self.checkpoint_engine = create_checkpoint_engine( + backend, + bucket_size_bytes=bucket_size_bytes, + engine_kwargs=engine_kwargs, + ) + + def prepare_checkpoint_engine(self) -> Any: # pragma: no cover + metadata = self.checkpoint_engine.prepare() + if isinstance(metadata, dict): + metadata = {**metadata, "rank": torch.distributed.get_rank()} + if self.checkpoint_engine.shard_expert_weights: + metadata["weight_layout"] = self._checkpoint_engine_weight_layout() + return metadata + + def init_checkpoint_engine_process_group( + self, + rank_prefix: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[Any], + ) -> None: # pragma: no cover + self.checkpoint_engine.init_rollout_process_group( + rollout_rank=resolve_rollout_rank(rank_prefix, rollout_world_size), + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + metadata=metadata, + ) + + def finalize_checkpoint_engine(self) -> None: # pragma: no cover + checkpoint_engine = getattr(self, "checkpoint_engine", None) + if checkpoint_engine is not None: + checkpoint_engine.finalize() + + async def _update_weights_from_checkpoint_engine_async(self) -> bool: + loaded_tensors = 0 + loaded_bytes = 0 + loaded_batches = 0 + load_time = 0.0 + start_time = time.time() + + async for weight_batch in self.checkpoint_engine.receive_weight_batches(): + loaded_batches += 1 + loaded_tensors += len(weight_batch) + loaded_bytes += sum(weight.nbytes for _name, weight in weight_batch) + + load_start = time.time() + self._load_weights(weight_batch) + torch.cuda.current_stream().synchronize() + load_time += time.time() - load_start + del weight_batch + + self._maybe_process_fp8_kv_cache() + + total_time = time.time() - start_time + loaded_gib = loaded_bytes / (1024 * 1024 * 1024) + print( + "[vLLM refit] Loaded " + f"{loaded_tensors} tensors in {loaded_batches} batches via checkpoint " + f"engine; bytes={loaded_gib:.2f}GiB total={total_time:.2f}s " + f"receive={max(total_time - load_time, 0.0):.2f}s load={load_time:.2f}s" + ) + return True + + @wrap_with_nvtx_name( + "vllm_internal_worker_extension/update_weights_from_checkpoint_engine" + ) + def update_weights_from_checkpoint_engine(self) -> bool: # pragma: no cover + return asyncio.run(self._update_weights_from_checkpoint_engine_async()) + + +class VllmCheckpointEngineRpcMixin: + """Dispatch checkpoint-engine calls through a synchronous vLLM engine.""" + + def checkpoint_engine_rpc( + self, checkpoint_method: str, method_args: tuple[Any, ...] = () + ) -> Any: # pragma: no cover + result = self.llm.collective_rpc(checkpoint_method, args=method_args) + if checkpoint_method == "update_weights_from_checkpoint_engine": + return all(item for item in result if item is not None) + return result + + +class VllmAsyncCheckpointEngineRpcMixin: + """Dispatch checkpoint-engine calls through an asynchronous vLLM engine.""" + + async def checkpoint_engine_rpc_async( + self, checkpoint_method: str, method_args: tuple[Any, ...] = () + ) -> Any: # pragma: no cover + from nemo_rl.models.generation.vllm.collective_rpc import ( + resolve_collective_rpc_result, + ) + + result = await self.llm.collective_rpc(checkpoint_method, args=method_args) + result = await resolve_collective_rpc_result(result) + if checkpoint_method == "update_weights_from_checkpoint_engine": + return all(item for item in result if item is not None) + return result diff --git a/nemo_rl/models/generation/vllm/collective_rpc.py b/nemo_rl/models/generation/vllm/collective_rpc.py new file mode 100644 index 00000000000..d8be66643fa --- /dev/null +++ b/nemo_rl/models/generation/vllm/collective_rpc.py @@ -0,0 +1,34 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import concurrent.futures +import inspect +from typing import Any + +import ray + + +async def resolve_collective_rpc_result(result: Any) -> Any: + """Recursively resolve a vLLM collective-RPC result.""" + while inspect.isawaitable(result): + result = await result + if isinstance(result, concurrent.futures.Future): + return await resolve_collective_rpc_result(await asyncio.wrap_future(result)) + if isinstance(result, ray.ObjectRef): + return await asyncio.to_thread(ray.get, result) + if isinstance(result, list | tuple): + items = [await resolve_collective_rpc_result(item) for item in result] + return tuple(items) if isinstance(result, tuple) else items + return result diff --git a/nemo_rl/models/generation/vllm/config.py b/nemo_rl/models/generation/vllm/config.py index 6b831cce923..03353836ca6 100644 --- a/nemo_rl/models/generation/vllm/config.py +++ b/nemo_rl/models/generation/vllm/config.py @@ -12,13 +12,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -from typing import Any, Literal, NotRequired, TypedDict +from typing import Annotated, Any, Literal, NotRequired, TypedDict, cast from pydantic import BaseModel, Field, NonNegativeInt, PositiveFloat, PositiveInt from nemo_rl.models.generation.interfaces import GenerationConfig VllmRefitTransportName = Literal["s3", "zmq"] +VllmRefitSelector = Literal["vllm_s3_sparse", "vllm_zmq_sparse", "nixl"] +VLLM_SPARSE_REFIT_TRANSPORTS = frozenset({"vllm_s3_sparse", "vllm_zmq_sparse"}) +VLLM_BUILTIN_REFIT_TRANSPORTS = VLLM_SPARSE_REFIT_TRANSPORTS | {"nixl"} class VllmSpecificArgs(TypedDict): @@ -100,7 +103,7 @@ class VllmRefitTuningConfig(BaseModel, extra="allow"): partition_workers: PositiveInt = 8 -class VllmRefitConfig(BaseModel, extra="allow"): +class VllmSparseRefitConfig(BaseModel, extra="allow"): delta_compression: VllmDeltaCompressionConfig = Field( default_factory=VllmDeltaCompressionConfig ) @@ -111,11 +114,32 @@ class VllmRefitConfig(BaseModel, extra="allow"): request_timeout_s: PositiveFloat = 600.0 +class VllmNixlRefitConfig(BaseModel, extra="forbid"): + update_weights_bucket_memory_ratio: Annotated[float, Field(gt=0, lt=1)] = 0.05 + device: str = "cuda" + backend_name: str = "UCX" + backend_init_params: dict[str, Any] | None = None + release_after_refit: bool = False + shard_expert_weights: bool = False + + +class VllmCheckpointEnginePluginConfig(BaseModel, extra="allow"): + update_weights_bucket_memory_ratio: Annotated[float, Field(gt=0, lt=1)] = 0.05 + release_after_refit: bool = False + + +class VllmRefitConfig(BaseModel, extra="allow"): + sparse: VllmSparseRefitConfig = Field(default_factory=VllmSparseRefitConfig) + nixl: VllmNixlRefitConfig = Field(default_factory=VllmNixlRefitConfig) + + class VllmConfig(GenerationConfig): vllm_cfg: VllmSpecificArgs vllm_kwargs: NotRequired[dict[str, Any]] - # Null uses NCCL; remote sparse refit supports S3 or ZeroMQ value planes. - refit_transport: NotRequired[Literal["vllm_s3_sparse", "vllm_zmq_sparse"] | None] + # Null uses the topology default (IPC colocated, NCCL non-colocated). + # Built-ins select sparse delta over S3/ZeroMQ or NIXL. + # A custom checkpoint engine may use a ``module:ClassName`` selector. + refit_transport: NotRequired[VllmRefitSelector | str | None] refit_cfg: NotRequired[VllmRefitConfig | None] # quantization config @@ -128,9 +152,30 @@ class VllmConfig(GenerationConfig): def normalize_vllm_refit_config(config: VllmConfig) -> VllmRefitConfig | None: - """Resolve sparse-refit defaults into the generation config.""" - if config.get("refit_transport") is None: + """Validate the selected refit transport and resolve its scoped defaults.""" + if cast(dict[str, Any], config).get("checkpoint_engine") is not None: + raise ValueError( + "policy.generation.checkpoint_engine was replaced by " + "policy.generation.refit_transport='nixl' and " + "policy.generation.refit_cfg.nixl." + ) + transport = config.get("refit_transport") + if transport is None: return None + if transport not in VLLM_BUILTIN_REFIT_TRANSPORTS and ":" not in transport: + raise ValueError( + f"Unknown vLLM refit transport {transport!r}: expected null, " + "'vllm_s3_sparse', 'vllm_zmq_sparse', 'nixl', or a " + "'module:ClassName' checkpoint-engine path." + ) refit_config = VllmRefitConfig.model_validate(config.get("refit_cfg") or {}) + if ":" in transport: + plugin_config = (refit_config.model_extra or {}).get(transport) + if plugin_config is None: + raise ValueError( + f"Custom checkpoint-engine transport {transport!r} requires " + f"policy.generation.refit_cfg[{transport!r}]." + ) + VllmCheckpointEnginePluginConfig.model_validate(plugin_config) config["refit_cfg"] = refit_config return refit_config diff --git a/nemo_rl/models/generation/vllm/patches.py b/nemo_rl/models/generation/vllm/patches.py index 1d7fbfbdbb0..f37c6fd94ec 100644 --- a/nemo_rl/models/generation/vllm/patches.py +++ b/nemo_rl/models/generation/vllm/patches.py @@ -283,7 +283,9 @@ def _patch_vllm_hermes_tool_parser_thread_safety(logger) -> None: def _apply_vllm_patches( - py_executable: str, *, extra_env_vars: list[str] | None = None + py_executable: str, + *, + extra_env_vars: list[str] | None = None, ) -> None: # Import lazily so importing the worker module does not import vLLM. from vllm.logger import init_logger diff --git a/nemo_rl/models/generation/vllm/refit_layout.py b/nemo_rl/models/generation/vllm/refit_layout.py new file mode 100644 index 00000000000..e2442fadde9 --- /dev/null +++ b/nemo_rl/models/generation/vllm/refit_layout.py @@ -0,0 +1,114 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Literal, TypedDict + +import torch + + +class VllmExpertParamLayout(TypedDict): + tp_rank: int + tp_size: int + local_expert_ids: list[int] | None + + +class VllmWeightLayout(TypedDict): + expert_params: dict[str, VllmExpertParamLayout] + missing_weight_prefixes: list[str] + + +@dataclass(frozen=True) +class HfExpertWeight: + parameter_name: str + expert_id: int + shard_id: Literal["w1", "w2", "w3"] + tp_shard_dim: int + + +_HF_EXPERT_WEIGHT_RE = re.compile( + r"^(?P.+\.mlp\.experts)\." + r"(?P\d+)\." + r"(?Pgate_proj|up_proj|down_proj)\.weight$" +) +_HF_PROJECTION_SHARDS: dict[str, Literal["w1", "w2", "w3"]] = { + "gate_proj": "w1", + "up_proj": "w3", + "down_proj": "w2", +} + + +def parse_hf_expert_weight(name: str) -> HfExpertWeight | None: + match = _HF_EXPERT_WEIGHT_RE.match(name) + if match is None: + return None + + projection = match.group("projection") + shard_id = _HF_PROJECTION_SHARDS[projection] + parameter_leaf = "w2_weight" if shard_id == "w2" else "w13_weight" + return HfExpertWeight( + parameter_name=f"{match.group('prefix')}.{parameter_leaf}", + expert_id=int(match.group("expert_id")), + shard_id=shard_id, + tp_shard_dim=1 if shard_id == "w2" else 0, + ) + + +def select_hf_weight_for_vllm_target( + name: str, + tensor: torch.Tensor, + *, + target_layout: VllmWeightLayout, +) -> torch.Tensor | None: + """Return the destination-local weight, or ``None`` if not owned. + + Pipeline stages omit complete parameter prefixes. Within an owned stage, + tensor-parallel MoE layers shard every expert tensor while expert-parallel + layers place complete experts on selected ranks. + """ + if any( + name.startswith(prefix) for prefix in target_layout["missing_weight_prefixes"] + ): + return None + + expert_weight = parse_hf_expert_weight(name) + if expert_weight is None: + return tensor + + param_layout = target_layout["expert_params"].get(expert_weight.parameter_name) + if param_layout is None: + # A missing expert parameter belongs to another pipeline stage. + return None + + local_expert_ids = param_layout["local_expert_ids"] + if local_expert_ids is not None and expert_weight.expert_id not in local_expert_ids: + return None + + tp_rank = param_layout["tp_rank"] + tp_size = param_layout["tp_size"] + + shard_dim = expert_weight.tp_shard_dim + if tp_size == 1: + return tensor + if tensor.shape[shard_dim] % tp_size != 0: + raise ValueError( + f"Cannot shard {name} dimension {shard_dim} of size " + f"{tensor.shape[shard_dim]} across vLLM TP size {tp_size}." + ) + + shard_size = tensor.shape[shard_dim] // tp_size + return tensor.narrow(shard_dim, tp_rank * shard_size, shard_size).contiguous() diff --git a/nemo_rl/models/generation/vllm/refit_loader.py b/nemo_rl/models/generation/vllm/refit_loader.py new file mode 100644 index 00000000000..7b2c0312faa --- /dev/null +++ b/nemo_rl/models/generation/vllm/refit_loader.py @@ -0,0 +1,263 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import cast + +import torch + +from nemo_rl.models.generation.vllm.refit_layout import ( + VllmExpertParamLayout, + VllmWeightLayout, + parse_hf_expert_weight, +) + + +class VllmShardedExpertRefitMixin: + """Load destination-local expert shards into vLLM storage.""" + + def _is_sharded_refit_weight(self, name: str, tensor: torch.Tensor) -> bool: + param_names = self._sharded_refit_param_names(name) + if not param_names: + return False + + state_dict_info = getattr(self, "state_dict_info", None) + if state_dict_info is None or name not in state_dict_info: + return False + full_shape, _dtype = state_dict_info[name] + return torch.Size(full_shape) != tensor.shape + + def _sharded_refit_param_names(self, name: str) -> list[str]: + expert_weight = parse_hf_expert_weight(name) + return [] if expert_weight is None else [expert_weight.parameter_name] + + def _checkpoint_engine_weight_layout(self) -> VllmWeightLayout: + from vllm.model_executor.models.utils import get_pp_missing_layer_names + + expert_params: dict[str, VllmExpertParamLayout] = {} + for name, param in self._get_named_parameters().items(): + if not name.endswith((".w13_weight", ".w2_weight")): + continue + + if bool(getattr(param, "is_transposed", False)): + raise ValueError( + "Sharded NIXL expert refit requires canonical expert-weight " + f"orientation, but {name} is transposed." + ) + + weight_loader = getattr(param, "weight_loader", None) + owner = getattr(weight_loader, "__self__", None) + if owner is None: + raise RuntimeError( + f"Could not inspect the vLLM expert weight loader for {name}." + ) + + self._validate_expert_storage(name, param) + + quant_method = getattr(owner, "base_quant_method", None) + backend = getattr(quant_method, "unquantized_backend", None) + backend_name = getattr(backend, "name", None) + if getattr(owner, "quant_config", None) is not None or backend_name not in { + "TRITON", + "BATCHED_TRITON", + }: + raise ValueError( + "Sharded NIXL expert refit requires canonical unquantized Triton " + f"expert weights, but {name} uses " + f"{type(quant_method).__name__}/{backend_name}. Set " + "policy.generation.vllm_kwargs.moe_backend=triton." + ) + + use_ep = bool(getattr(owner, "use_ep", False)) + local_expert_ids: list[int] | None = None + if use_ep: + if bool(getattr(owner, "enable_eplb", False)): + raise RuntimeError( + "Sharded refit does not support dynamic vLLM expert load " + "balancing because ownership can change after metadata " + "exchange." + ) + expert_map = getattr(owner, "_expert_map", None) + if expert_map is None: + raise RuntimeError( + f"vLLM reports EP for {name} without an expert ownership map." + ) + logical_num_experts = int( + cast( + int, + getattr(owner, "logical_num_experts", expert_map.numel()), + ) + ) + global_num_experts = int( + cast( + int, + getattr(owner, "global_num_experts", logical_num_experts), + ) + ) + if global_num_experts != logical_num_experts: + raise RuntimeError( + "Sharded refit does not support redundant vLLM experts." + ) + local_expert_ids = [ + expert_id + for expert_id, local_id in enumerate( + expert_map.detach().cpu().tolist()[:logical_num_experts] + ) + if int(local_id) >= 0 + ] + + expert_params[name] = { + "tp_rank": int(getattr(owner, "tp_rank", 0)), + "tp_size": int(getattr(owner, "tp_size", 1)), + "local_expert_ids": local_expert_ids, + } + + return { + "expert_params": expert_params, + "missing_weight_prefixes": get_pp_missing_layer_names( + self.model_runner.model + ), + } + + @staticmethod + def _validate_expert_storage(param_name: str, param: torch.nn.Parameter) -> None: + """Reject incompatible vLLM storage before advertising shards.""" + is_w13 = param_name.endswith(".w13_weight") + if param.ndim != 3 or (is_w13 and param.shape[1] % 2 != 0): + raise RuntimeError( + f"Unsupported vLLM expert storage for {param_name}: " + f"shape={tuple(param.shape)}." + ) + + def _local_expert_id(self, param: torch.nn.Parameter, expert_id: int) -> int: + weight_loader = getattr(param, "weight_loader", None) + owner = getattr(weight_loader, "__self__", None) + mapper = getattr(owner, "_map_global_expert_id_to_local_expert_id", None) + if mapper is None: + return expert_id + return int(mapper(expert_id)) + + def _load_destination_local_expert_group( + self, + param_name: str, + param: torch.nn.Parameter, + shard_id: str, + items: list[tuple[int, torch.Tensor]], + ) -> None: + """Copy destination-local experts into canonical vLLM storage.""" + if shard_id not in {"w1", "w2", "w3"}: + raise ValueError(f"Unexpected sharded expert shard_id: {shard_id}") + + target = param.data + if shard_id in {"w1", "w3"}: + shard_size = target.shape[1] // 2 + shard_offset = 0 if shard_id == "w1" else shard_size + target = target.narrow(1, shard_offset, shard_size) + + def copy_to_target(destination: torch.Tensor, source: torch.Tensor) -> None: + for dim, size in enumerate(source.shape): + destination = destination.narrow(dim, 0, size) + destination.copy_(source) + + sorted_items = sorted(items, key=lambda item: item[0]) + expert_ids = [expert_id for expert_id, _tensor in sorted_items] + contiguous_ids = list(range(expert_ids[0], expert_ids[0] + len(expert_ids))) + with torch.no_grad(): + if expert_ids == contiguous_ids: + expert_data = target.narrow(0, expert_ids[0], len(expert_ids)) + loaded_weight = torch.stack( + [tensor for _expert_id, tensor in sorted_items] + ) + copy_to_target(expert_data, loaded_weight) + else: + for local_expert_id, loaded_weight in sorted_items: + copy_to_target(target[local_expert_id], loaded_weight) + + def _load_sharded_expert_weight_groups( + self, weights: list[tuple[str, torch.Tensor]] + ) -> list[tuple[str, torch.Tensor]]: + params = self._get_named_parameters() + groups: dict[tuple[str, str], list[tuple[int, torch.Tensor]]] = {} + remaining_weights: list[tuple[str, torch.Tensor]] = [] + + for name, tensor in weights: + expert_weight = parse_hf_expert_weight(name) + if expert_weight is None: + remaining_weights.append((name, tensor)) + continue + + mapped_name = expert_weight.parameter_name + param = params.get(mapped_name) + if param is None: + remaining_weights.append((name, tensor)) + continue + + owner = getattr(getattr(param, "weight_loader", None), "__self__", None) + if not self._is_sharded_refit_weight(name, tensor) and not bool( + getattr(owner, "use_ep", False) + ): + remaining_weights.append((name, tensor)) + continue + + if owner is None: + raise RuntimeError( + "Could not resolve the vLLM expert weight loader for " + f"{mapped_name}." + ) + + if param.data.ndim != 3 or tensor.ndim != 2: + raise ValueError( + f"Sharded expert {name} requires a 3-D vLLM parameter and " + f"2-D source tensor, got {param.data.ndim}-D and {tensor.ndim}-D." + ) + + full_shape, _dtype = self.state_dict_info[name] + expected_shape = list(full_shape) + tp_size = int(getattr(owner, "tp_size", 1)) + shard_dim = expert_weight.tp_shard_dim + expected_shape[shard_dim] //= tp_size + if tensor.shape != torch.Size(expected_shape): + raise ValueError( + f"Received sharded expert {name} with shape " + f"{tuple(tensor.shape)}, expected {tuple(expected_shape)} from " + f"full shape {tuple(full_shape)} and TP size {tp_size}." + ) + + local_expert_id = self._local_expert_id(param, expert_weight.expert_id) + if local_expert_id == -1: + continue + + groups.setdefault((mapped_name, expert_weight.shard_id), []).append( + (local_expert_id, tensor) + ) + + for (mapped_name, shard_id), items in groups.items(): + self._load_destination_local_expert_group( + mapped_name, params[mapped_name], shard_id, items + ) + + return remaining_weights + + def _load_sharded_expert_weights( + self, policy_weights: list[tuple[str, torch.Tensor]] + ) -> None: + from nemo_rl.models.generation.vllm.quantization import fp8 + + if fp8.is_fp8_model(self.model_runner.vllm_config): + raise ValueError( + "Sharded NIXL expert refit is not supported for FP8 vLLM models." + ) + + remaining_weights = self._load_sharded_expert_weight_groups(policy_weights) + if remaining_weights: + self._load_full_hf_weights(remaining_weights) diff --git a/nemo_rl/models/generation/vllm/vllm_backend.py b/nemo_rl/models/generation/vllm/vllm_backend.py index 7b4310ce9b2..d21f169a261 100644 --- a/nemo_rl/models/generation/vllm/vllm_backend.py +++ b/nemo_rl/models/generation/vllm/vllm_backend.py @@ -22,6 +22,11 @@ import torch import zmq +from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmCheckpointEngineMixin, + preinit_nixl_from_vllm_config, + resolve_rollout_rank, +) from nemo_rl.models.policy.utils import ( IPCProtocol, calculate_aligned_size, @@ -32,6 +37,7 @@ try: import vllm # noqa: F401 + from vllm.v1.worker.gpu_worker import Worker as VllmWorker except ImportError: raise ImportError( "vLLM is not installed. Please check that the py_executable in the runtime_env of VllmGenerationWorker " @@ -101,6 +107,15 @@ def require_complete(self) -> None: raise IPCWeightManifestError("; ".join(details)) +class NixlVllmWorker(VllmWorker): + """vLLM worker that establishes NIXL/UCX before vLLM initialization.""" + + def __new__(cls, vllm_config: Any, *args: Any, **kwargs: Any) -> "NixlVllmWorker": + worker = super().__new__(cls) + worker._nrl_nixl_preinit_agent = preinit_nixl_from_vllm_config(vllm_config) + return worker + + def fix_gemma3_vision_weight_name(key: str) -> str: """Re-insert the `vision_model` segment into Gemma3 vision-tower weights. @@ -156,6 +171,27 @@ def _read_mtp_layer_weights_from_checkpoint( class VllmInternalWorkerExtension: _sparse_delta_applier: Any = None + _nrl_named_parameters: dict[str, torch.nn.Parameter] + + def _get_named_parameters(self) -> dict[str, torch.nn.Parameter]: + params = getattr(self, "_nrl_named_parameters", None) + if params is None: + params = dict(self.model_runner.model.named_parameters()) + self._nrl_named_parameters = params + return params + + def _load_full_hf_weights( + self, policy_weights: list[tuple[str, torch.Tensor]] + ) -> None: + self.model_runner.model.load_weights(weights=policy_weights) + + def _load_hf_weights(self, policy_weights: list[tuple[str, torch.Tensor]]) -> None: + from nemo_rl.models.generation.vllm.quantization import fp8 + + if fp8.is_fp8_model(self.model_runner.vllm_config): + fp8.load_weights(policy_weights, self.model_runner) + return + self._load_full_hf_weights(policy_weights) def bind_numa(self) -> bool: """Pin this TP worker to its GPU's NUMA-local CPUs/memory. @@ -187,9 +223,10 @@ def init_collective( """Initialize the collective communication.""" from nemo_rl.distributed.stateless_process_group import StatelessProcessGroup - local_rank = torch.distributed.get_rank() # Place vLLM ranks after all training ranks so all training workers can join - rank = train_world_size + rank_prefix + local_rank + rank = train_world_size + resolve_rollout_rank( + rank_prefix, world_size - train_world_size + ) self.model_update_group = StatelessProcessGroup( # pyrefly: ignore[implicitly-defined-attribute] This class does not define __init__ so assignments like this should be ignored master_address=ip, port=port, rank=rank, world_size=world_size @@ -407,11 +444,9 @@ def _load_weights(self, weights): """Load weights with Gemma3 vision-tower weight name fix, FP8, and draft-weight support. Applies Gemma3 vision-tower weight name fix if needed, splits policy/draft - weights, applies FP8 conversion if needed, and loads draft weights - into the drafter model. + weights, dispatches policy weights through the configured refit loader, + and loads draft weights into the drafter model. """ - from nemo_rl.models.generation.vllm.quantization import fp8 - if ( "Gemma3ForConditionalGeneration" in self.model_runner.vllm_config.model_config.architectures @@ -420,11 +455,7 @@ def _load_weights(self, weights): weights[idx] = (fix_gemma3_vision_weight_name(key), weight) policy_weights, draft_weights = self._split_policy_and_draft_weights(weights) - if fp8.is_fp8_model(self.model_runner.vllm_config): - fp8.load_weights(policy_weights, self.model_runner) - else: - self.model_runner.model.load_weights(weights=policy_weights) - + self._load_hf_weights(policy_weights) self._load_draft_weights(draft_weights) def _get_sparse_delta_applier(self) -> Any: @@ -626,3 +657,9 @@ def start_gpu_profiling(self) -> None: def stop_gpu_profiling(self) -> None: """Stop GPU profiling.""" torch.cuda.profiler.stop() + + +class VllmInternalWorkerExtensionWithCheckpointEngine( + VllmCheckpointEngineMixin, VllmInternalWorkerExtension +): + """vLLM worker extension with checkpoint-engine refit support.""" diff --git a/nemo_rl/models/generation/vllm/vllm_sparse_refit.py b/nemo_rl/models/generation/vllm/vllm_sparse_refit.py index b298d0674d1..ee8f42f163c 100644 --- a/nemo_rl/models/generation/vllm/vllm_sparse_refit.py +++ b/nemo_rl/models/generation/vllm/vllm_sparse_refit.py @@ -36,7 +36,10 @@ _get_free_port_local, _get_node_ip_local, ) -from nemo_rl.models.generation.vllm.config import VllmRefitConfig +from nemo_rl.models.generation.vllm.config import ( + VllmRefitConfig, + VllmSparseRefitConfig, +) from nemo_rl.utils import weight_transfer_sparse_codec as sparse_codec from nemo_rl.utils.weight_transfer_http import ( G_VLLM_REFIT_API_KEY_HEADER, @@ -104,9 +107,8 @@ class VllmSparseRefitReceiver: def __init__(self, worker: Any) -> None: self._worker = worker - self._refit_config = VllmRefitConfig.model_validate( - worker.cfg.get("refit_cfg") or {} - ) + refit_config = VllmRefitConfig.model_validate(worker.cfg.get("refit_cfg") or {}) + self._refit_config: VllmSparseRefitConfig = refit_config.sparse tuning = self._refit_config.tuning self._refit_apply_queue_condition = threading.Condition() self._refit_apply_executor = ThreadPoolExecutor( diff --git a/nemo_rl/models/generation/vllm/vllm_worker.py b/nemo_rl/models/generation/vllm/vllm_worker.py index b3a8aa2b63d..3491a14ebff 100644 --- a/nemo_rl/models/generation/vllm/vllm_worker.py +++ b/nemo_rl/models/generation/vllm/vllm_worker.py @@ -37,16 +37,29 @@ resolve_routed_experts_dtype, verify_right_padding, ) -from nemo_rl.models.generation.vllm.config import VllmConfig +from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmCheckpointEngineRpcMixin, +) +from nemo_rl.models.generation.vllm.config import ( + VLLM_SPARSE_REFIT_TRANSPORTS, + VllmConfig, +) from nemo_rl.models.generation.vllm.patches import _apply_vllm_patches from nemo_rl.models.generation.vllm.utils import ( format_prompt_for_vllm_generation, pad_and_align_routed_expert_indices, ) +from nemo_rl.models.generation.vllm.worker_utils import ( + resolve_data_parallel_local_rank, + resolve_distributed_executor_backend, +) from nemo_rl.models.huggingface.common import ModelFlag from nemo_rl.models.policy.utils import is_vllm_v1_engine_enabled from nemo_rl.utils.nsys import wrap_with_nvtx_name from nemo_rl.utils.nvml import log_gpu_memory_diagnostics +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) logger = logging.getLogger(__name__) @@ -248,8 +261,11 @@ def __init__( config, bundle_indices, fraction_of_gpus, seed, extra_env_vars ) self._sparse_refit_receiver: Any = None - if self.is_model_owner and self.cfg.get("refit_transport") is not None: - # Avoid receiver dependencies and threads for existing refit transports. + if ( + self.is_model_owner + and self.cfg.get("refit_transport") in VLLM_SPARSE_REFIT_TRANSPORTS + ): + # Keep sparse receiver dependencies and threads off other refit paths. from nemo_rl.models.generation.vllm.vllm_sparse_refit import ( VllmSparseRefitReceiver, ) @@ -288,7 +304,10 @@ def _init_config( # Store the Python executable being used by this worker self.py_executable = sys.executable - _apply_vllm_patches(self.py_executable, extra_env_vars=extra_env_vars) + _apply_vllm_patches( + self.py_executable, + extra_env_vars=extra_env_vars, + ) # Skip model loading if we're not the model owner if not self.is_model_owner: @@ -328,6 +347,13 @@ def _load_model(self, bundle_indices, seed): "please run at least once with the environment variable NRL_FORCE_REBUILD_VENVS=true set to force the rebuild of the environment." ) vllm_kwargs: dict[str, Any] = copy.deepcopy(self.cfg.get("vllm_kwargs", {})) + checkpoint_engine_config = checkpoint_engine_refit_config(self.cfg) + if checkpoint_engine_config is not None: + from nemo_rl.models.generation.vllm.checkpoint_engine import ( + configure_nixl_worker, + ) + + configure_nixl_worker(self.cfg, vllm_kwargs) # Calculate total parallel size (TP * PP) model_parallel_size = self.tensor_parallel_size * self.pipeline_parallel_size @@ -348,11 +374,12 @@ def _load_model(self, bundle_indices, seed): f"VLLM_RAY_BUNDLE_INDICES environment variable set to: {os.environ.get('VLLM_RAY_BUNDLE_INDICES')}" ) - # Use Ray for distributed execution in parallel mode - vllm_kwargs["distributed_executor_backend"] = "ray" - else: - # For non-parallel mode, explicitly set executor to None to avoid Ray issues - vllm_kwargs["distributed_executor_backend"] = None + executor_backend = resolve_distributed_executor_backend( + self.tensor_parallel_size, + self.pipeline_parallel_size, + self.expert_parallel_size, + ) + vllm_kwargs["distributed_executor_backend"] = executor_backend os.environ["VLLM_USE_V1"] = "1" if is_vllm_v1_engine_enabled() else "0" os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1" @@ -364,7 +391,11 @@ def _load_model(self, bundle_indices, seed): world_size = int(os.environ["VLLM_DP_SIZE"]) * model_parallel_size rank = int(os.environ["RANK"]) % world_size os.environ["VLLM_DP_RANK"] = str(rank // model_parallel_size) - os.environ["VLLM_DP_RANK_LOCAL"] = str((rank % 8) // model_parallel_size) + os.environ["VLLM_DP_RANK_LOCAL"] = str( + resolve_data_parallel_local_rank( + rank, model_parallel_size, executor_backend + ) + ) # set vLLM DP address and port leader_rank = int(os.environ["RANK"]) // world_size * world_size addr_list = eval(os.environ["AVAILABLE_ADDR_LIST"]) @@ -512,7 +543,13 @@ def _load_model(self, bundle_indices, seed): enforce_eager=self.cfg["vllm_cfg"]["enforce_eager"], max_model_len=self.cfg["vllm_cfg"]["max_model_len"], trust_remote_code=True, - worker_extension_cls="nemo_rl.models.generation.vllm.vllm_backend.VllmInternalWorkerExtension", + worker_extension_cls=( + "nemo_rl.models.generation.vllm.vllm_backend." + "VllmInternalWorkerExtensionWithCheckpointEngine" + if checkpoint_engine_config is not None + else "nemo_rl.models.generation.vllm.vllm_backend." + "VllmInternalWorkerExtension" + ), enable_sleep_mode=True, # Set disable_log_stats=False so that self.llm.get_metrics() works. disable_log_stats=False, @@ -680,7 +717,7 @@ def stop_zmq_sparse_refit_relay(self) -> None: receiver.stop_zmq_sparse_refit_relay() -class VllmGenerationWorkerImpl(BaseVllmGenerationWorker): +class VllmGenerationWorkerImpl(VllmCheckpointEngineRpcMixin, BaseVllmGenerationWorker): def _create_engine(self, llm_kwargs: dict[str, Any]) -> None: import vllm diff --git a/nemo_rl/models/generation/vllm/vllm_worker_async.py b/nemo_rl/models/generation/vllm/vllm_worker_async.py index 2f50e4a150e..b08184196b9 100644 --- a/nemo_rl/models/generation/vllm/vllm_worker_async.py +++ b/nemo_rl/models/generation/vllm/vllm_worker_async.py @@ -40,6 +40,9 @@ GenerationOutputSpec, verify_right_padding, ) +from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmAsyncCheckpointEngineRpcMixin, +) from nemo_rl.models.generation.vllm.utils import ( attach_routed_experts_to_chat_response_choices, format_prompt_for_vllm_generation, @@ -157,7 +160,9 @@ def _replace_prefix_tokens( ) -class VllmAsyncGenerationWorkerImpl(BaseVllmGenerationWorker): +class VllmAsyncGenerationWorkerImpl( + VllmAsyncCheckpointEngineRpcMixin, BaseVllmGenerationWorker +): def __init__( self, config, diff --git a/nemo_rl/models/generation/vllm/worker_utils.py b/nemo_rl/models/generation/vllm/worker_utils.py new file mode 100644 index 00000000000..831fbde0dd5 --- /dev/null +++ b/nemo_rl/models/generation/vllm/worker_utils.py @@ -0,0 +1,35 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def resolve_distributed_executor_backend( + tensor_parallel_size: int, + pipeline_parallel_size: int, + expert_parallel_size: int, +) -> str | None: + if tensor_parallel_size * pipeline_parallel_size > 1: + return "ray" + if expert_parallel_size > tensor_parallel_size: + # External DP actors already own one GPU each. + return "uni" + return None + + +def resolve_data_parallel_local_rank( + rank: int, model_parallel_size: int, executor_backend: str | None +) -> int: + # Ray remaps one GPU into each external-DP actor. + if executor_backend == "uni": + return 0 + return (rank % 8) // model_parallel_size diff --git a/nemo_rl/models/policy/workers/checkpoint_engine.py b/nemo_rl/models/policy/workers/checkpoint_engine.py new file mode 100644 index 00000000000..85e7fdd4c72 --- /dev/null +++ b/nemo_rl/models/policy/workers/checkpoint_engine.py @@ -0,0 +1,173 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import warnings +from collections.abc import Generator, Iterator +from typing import TYPE_CHECKING, Any, Optional, cast + +import torch + +from nemo_rl.models.generation.vllm.refit_layout import ( + VllmWeightLayout, + select_hf_weight_for_vllm_target, +) +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) + +if TYPE_CHECKING: + from nemo_rl.utils.checkpoint_engines.base import CheckpointEngine + + +def maybe_preinit_nixl_checkpoint_engine(config: dict[str, Any]) -> Any: + """Preinitialize NIXL when checkpoint-engine refit is configured.""" + generation_config = config.get("generation") + if generation_config is None: + return None + checkpoint_config = checkpoint_engine_refit_config(generation_config) + if checkpoint_config is None or checkpoint_config["backend"] != "nixl": + return None + + from nemo_rl.utils.checkpoint_engines.nixl import ( + preinit_nixl_agent, + resolve_nixl_backend_kwargs, + ) + + backend_name, backend_init_params = resolve_nixl_backend_kwargs( + checkpoint_config["engine_kwargs"]["nixl"] + ) + return preinit_nixl_agent( + backend_name=backend_name, backend_init_params=backend_init_params + ) + + +class DTensorCheckpointEngineSendMixin: + """Onload DTensor/FSDP2 policy weights for checkpoint-engine transfer.""" + + model: torch.nn.Module + + def _prepare_checkpoint_engine_weight_send(self) -> None: + if self.cpu_offload: + warnings.warn( + "cpu_offload adds an onload/offload cycle during non-colocated " + "checkpoint-engine refit. Disable it unless GPU memory requires it.", + stacklevel=2, + ) + self.model = self.move_to_cuda(self.model) + + def _finalize_checkpoint_engine_weight_send(self) -> None: + if self.cpu_offload: + self.model = self.move_to_cpu(self.model) + + def _checkpoint_engine_weight_iterator( + self, kv_scales: Optional[dict[str, float]] = None + ) -> Iterator[tuple[str, torch.Tensor]]: + if kv_scales is not None: + raise NotImplementedError( + "FP8 kvcache is not currently supported for DTensor path, we will support it in the future." + ) + return self._checkpoint_engine_params() + + +class MegatronCheckpointEngineSendMixin: + """Select destination-local Megatron weights for checkpoint-engine transfer.""" + + def _checkpoint_engine_weight_iterator( + self, kv_scales: Optional[dict[str, float]] = None + ) -> Iterator[tuple[str, torch.Tensor]]: + weights = self._iter_params_with_optional_kv_scales(kv_scales=kv_scales) + target_layout = self.checkpoint_engine.get_target_weight_layout() + if target_layout is None: + return weights + + target_layout = cast(VllmWeightLayout, target_layout) + return ( + (name, selected) + for name, tensor in weights + if ( + selected := select_hf_weight_for_vllm_target( + name, tensor, target_layout=target_layout + ) + ) + is not None + ) + + +class PolicyCheckpointEngineMixin: + """Checkpoint-engine lifecycle shared by policy worker implementations.""" + + checkpoint_engine: "CheckpointEngine" + rank: int + + def _checkpoint_engine_weight_iterator( + self, kv_scales: Optional[dict[str, float]] = None + ) -> Generator[tuple[str, torch.Tensor], None, None]: + raise NotImplementedError( + f"{self.__class__.__name__} does not support checkpoint-engine refit." + ) + + def _prepare_checkpoint_engine_weight_send(self) -> None: + pass + + def _finalize_checkpoint_engine_weight_send(self) -> None: + pass + + async def send_weights_via_checkpoint_engine( + self, kv_scales: Optional[dict[str, float]] = None + ) -> None: + self._prepare_checkpoint_engine_weight_send() + try: + with torch.no_grad(): + await self.checkpoint_engine.send_weights( + self._checkpoint_engine_weight_iterator(kv_scales=kv_scales) + ) + finally: + self._finalize_checkpoint_engine_weight_send() + + async def checkpoint_engine_rpc( + self, checkpoint_method: str, method_kwargs: Optional[dict[str, Any]] = None + ) -> Any: + kwargs = method_kwargs or {} + if checkpoint_method == "checkpoint_engine_total_memory_bytes": + device = torch.cuda.current_device() + return torch.cuda.get_device_properties(device).total_memory + if checkpoint_method == "init_checkpoint_engine": + if getattr(self, "checkpoint_engine", None) is None: + from nemo_rl.utils.checkpoint_engines.base import ( + create_checkpoint_engine, + ) + + self.checkpoint_engine = create_checkpoint_engine( + kwargs["backend"], + bucket_size_bytes=kwargs["bucket_size_bytes"], + engine_kwargs=kwargs["engine_kwargs"], + ) + return + if checkpoint_method == "prepare_checkpoint_engine": + metadata = self.checkpoint_engine.prepare() + if isinstance(metadata, dict): + return {**metadata, "rank": self.rank} + return metadata + if checkpoint_method == "init_checkpoint_engine_process_group": + return self.checkpoint_engine.init_policy_process_group( + worker_rank=self.rank, **kwargs + ) + if checkpoint_method == "send_weights_via_checkpoint_engine": + return await self.send_weights_via_checkpoint_engine(**kwargs) + if checkpoint_method == "finalize_checkpoint_engine": + checkpoint_engine = getattr(self, "checkpoint_engine", None) + if checkpoint_engine is not None: + checkpoint_engine.finalize() + return + return getattr(self.checkpoint_engine, checkpoint_method)(**kwargs) diff --git a/nemo_rl/models/policy/workers/dtensor_policy_worker.py b/nemo_rl/models/policy/workers/dtensor_policy_worker.py index 0083e5301e2..d132d1c111f 100644 --- a/nemo_rl/models/policy/workers/dtensor_policy_worker.py +++ b/nemo_rl/models/policy/workers/dtensor_policy_worker.py @@ -87,6 +87,11 @@ resolve_model_class, ) from nemo_rl.models.policy.workers.base_policy_worker import AbstractPolicyWorker +from nemo_rl.models.policy.workers.checkpoint_engine import ( + DTensorCheckpointEngineSendMixin, + PolicyCheckpointEngineMixin, + maybe_preinit_nixl_checkpoint_engine, +) from nemo_rl.utils.grad_norm import warn_if_inf_grad_norm from nemo_rl.utils.native_checkpoint import ( load_checkpoint, @@ -97,6 +102,15 @@ from nemo_rl.utils.timer import Timer +def dtensor_params_generator( + model: nn.Module, target_dtype: torch.dtype +) -> Generator[tuple[str, torch.Tensor], None, None]: + """Yield contiguous full tensors from a DTensor-backed state dict.""" + for name, tensor in model.state_dict().items(): + full_tensor = tensor.full_tensor() if isinstance(tensor, DTensor) else tensor + yield name, full_tensor.to(target_dtype, non_blocking=True).contiguous() + + def _attach_context_parallel_hooks(model: nn.Module) -> None: """Attach forward pre-hooks to self_attn modules for context parallelism. @@ -168,7 +182,11 @@ def get_cpu_state_dict( # Classes with @ray.remote can't be inherited from, so we split the implementation out. # This is useful when using worker extension classes. class DTensorPolicyWorkerImpl( - TQWorkerMixin, AbstractPolicyWorker, ColocatablePolicyInterface + TQWorkerMixin, + DTensorCheckpointEngineSendMixin, + PolicyCheckpointEngineMixin, + AbstractPolicyWorker, + ColocatablePolicyInterface, ): def __repr__(self) -> str: """Customizes the actor's prefix in the Ray logs. @@ -402,6 +420,7 @@ def __init__( self.tp_size = tp_size self.cp_size = cp_size self.device_mesh = device_mesh + self._nixl_preinit_agent = maybe_preinit_nixl_checkpoint_engine(config) # ------------------------------------------------ # 3) Move to GPU + Composable FSDP @@ -1859,30 +1878,20 @@ def stream_weights_via_ipc_zmq( from nemo_rl.models.policy.utils import stream_weights_via_ipc_zmq_impl - def dtensor_params_generator(): - """Generator that yields (name, tensor) pairs, converting DTensors to local tensors.""" - for name, tensor in self.model.state_dict().items(): - if isinstance(tensor, DTensor): - # Convert DTensor to full tensor for streaming - full_tensor = tensor.full_tensor() - # Convert to target dtype - yield ( - name, - full_tensor.to(self.dtype, non_blocking=True).contiguous(), - ) - else: - # Convert to target dtype - yield name, tensor.to(self.dtype, non_blocking=True).contiguous() - # Use the shared implementation stream_weights_via_ipc_zmq_impl( - params_generator=dtensor_params_generator(), + params_generator=dtensor_params_generator(self.model, self.dtype), buffer_size_bytes=buffer_size_bytes, zmq_socket=self.zmq_socket, rank=self.rank, worker_name=str(self), ) + def _checkpoint_engine_params( + self, + ) -> Generator[tuple[str, torch.Tensor], None, None]: + return dtensor_params_generator(self.model, self.dtype) + @torch.no_grad() def broadcast_weights_for_collective( self, kv_scales: Optional[dict[str, float]] = None diff --git a/nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py b/nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py index 12a01b2a6be..8ed9d584fed 100644 --- a/nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py +++ b/nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py @@ -72,6 +72,11 @@ get_runtime_env_for_policy_worker, ) from nemo_rl.models.policy.workers.base_policy_worker import AbstractPolicyWorker +from nemo_rl.models.policy.workers.checkpoint_engine import ( + DTensorCheckpointEngineSendMixin, + PolicyCheckpointEngineMixin, + maybe_preinit_nixl_checkpoint_engine, +) from nemo_rl.models.policy.workers.patches import ( apply_transformer_engine_patch, ) @@ -198,7 +203,11 @@ def get_train_context( # Classes with @ray.remote can't be inherited from, so we split the implementation out. # This is useful when using worker extension classes. class DTensorPolicyWorkerV2Impl( - TQWorkerMixin, AbstractPolicyWorker, ColocatablePolicyInterface + TQWorkerMixin, + DTensorCheckpointEngineSendMixin, + PolicyCheckpointEngineMixin, + AbstractPolicyWorker, + ColocatablePolicyInterface, ): def __repr__(self) -> str: """Customizes the actor's prefix in the Ray logs. @@ -300,6 +309,7 @@ def __init__( self.dp_size = distributed_context.dp_size self.tp_size = distributed_context.tp_size self.cp_size = distributed_context.cp_size + self._nixl_preinit_agent = maybe_preinit_nixl_checkpoint_engine(config) # Initialize checkpoint manager now that distributed is set up self._init_checkpoint_manager( @@ -1164,6 +1174,11 @@ def stream_weights_via_http( worker_state=self._ipc_worker_state, ) + def _checkpoint_engine_params( + self, + ) -> Generator[tuple[str, torch.Tensor], None, None]: + return dtensor_params_generator(self.model, self.dtype) + @torch.no_grad() def broadcast_weights_for_collective( self, kv_scales: Optional[dict[str, float]] = None diff --git a/nemo_rl/models/policy/workers/megatron_policy_worker.py b/nemo_rl/models/policy/workers/megatron_policy_worker.py index cf0e2621ed3..eecd310a782 100644 --- a/nemo_rl/models/policy/workers/megatron_policy_worker.py +++ b/nemo_rl/models/policy/workers/megatron_policy_worker.py @@ -92,6 +92,11 @@ ) from nemo_rl.models.policy.utils import get_runtime_env_for_policy_worker from nemo_rl.models.policy.workers.base_policy_worker import AbstractPolicyWorker +from nemo_rl.models.policy.workers.checkpoint_engine import ( + MegatronCheckpointEngineSendMixin, + PolicyCheckpointEngineMixin, + maybe_preinit_nixl_checkpoint_engine, +) from nemo_rl.models.policy.workers.patches import apply_transformer_engine_patch from nemo_rl.utils.grad_norm import warn_if_inf_grad_norm from nemo_rl.utils.nsys import wrap_with_nvtx_name @@ -146,6 +151,8 @@ class MegatronPolicyWorkerImpl( MegatronGenerationMixin, MegatronGenerationRefitMixin, TQWorkerMixin, + MegatronCheckpointEngineSendMixin, + PolicyCheckpointEngineMixin, AbstractPolicyWorker, ColocatablePolicyInterface, ): @@ -294,6 +301,7 @@ def __init__( self.cfg = config self._router_replay_enabled = router_replay_enabled(config) + self._nixl_preinit_agent = maybe_preinit_nixl_checkpoint_engine(config) # Set rank for non-collocated to check which ranks to broadcast from self.rank = get_rank_safe() @@ -1846,7 +1854,9 @@ def _require_remote_sparse_refit(self) -> Any: refit_config = self.cfg["generation"]["refit_cfg"] assert refit_config is not None - self._remote_sparse_refit = MegatronRemoteSparseRefit(self, refit_config) + self._remote_sparse_refit = MegatronRemoteSparseRefit( + self, refit_config.sparse + ) return self._remote_sparse_refit def finish_remote_sparse_delta_sync(self, *, succeeded: bool) -> None: diff --git a/nemo_rl/models/policy/workers/megatron_remote_sparse_refit.py b/nemo_rl/models/policy/workers/megatron_remote_sparse_refit.py index dced0511ee0..b9ab86fc04b 100644 --- a/nemo_rl/models/policy/workers/megatron_remote_sparse_refit.py +++ b/nemo_rl/models/policy/workers/megatron_remote_sparse_refit.py @@ -19,8 +19,8 @@ import torch from nemo_rl.models.generation.vllm.config import ( - VllmRefitConfig, VllmRefitTransportName, + VllmSparseRefitConfig, ) from nemo_rl.utils.weight_transfer_sparse_codec import DeltaCompressionTracker from nemo_rl.utils.weight_transfer_stream import ( @@ -31,7 +31,7 @@ class MegatronRemoteSparseRefit: - def __init__(self, worker: Any, refit_config: VllmRefitConfig) -> None: + def __init__(self, worker: Any, refit_config: VllmSparseRefitConfig) -> None: self._worker = worker self._tracker = DeltaCompressionTracker(refit_config) diff --git a/nemo_rl/utils/checkpoint_engines/__init__.py b/nemo_rl/utils/checkpoint_engines/__init__.py new file mode 100644 index 00000000000..4fc25d0d3c9 --- /dev/null +++ b/nemo_rl/utils/checkpoint_engines/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nemo_rl/utils/checkpoint_engines/base.py b/nemo_rl/utils/checkpoint_engines/base.py new file mode 100644 index 00000000000..721106c8224 --- /dev/null +++ b/nemo_rl/utils/checkpoint_engines/base.py @@ -0,0 +1,162 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import importlib +from abc import ABC, abstractmethod +from collections.abc import AsyncGenerator, Generator +from dataclasses import dataclass +from typing import Any + +import torch + + +@dataclass +class TensorMeta: + name: str + shape: torch.Size + dtype: torch.dtype + chunk_offset: int + chunk_size: int + offset: int | None + + @property + def nbytes(self) -> int: + return self.shape.numel() * self.dtype.itemsize + + +class CheckpointEngine(ABC): + shard_expert_weights: bool = False + + def get_target_weight_layout(self) -> dict[str, Any] | None: + """Return the destination-local layout for this policy rank, if any.""" + if self.shard_expert_weights: + raise NotImplementedError( + f"{type(self).__name__} must implement get_target_weight_layout() " + "when shard_expert_weights is enabled." + ) + return None + + @abstractmethod + def prepare(self) -> Any: + """Allocate or register backend resources and return serializable metadata.""" + raise NotImplementedError + + @abstractmethod + def init_policy_process_group( + self, + *, + worker_rank: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[Any], + ) -> None: + """Connect a policy worker to its transfer peer.""" + raise NotImplementedError + + @abstractmethod + def init_rollout_process_group( + self, + *, + rollout_rank: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[Any], + ) -> None: + """Connect a rollout worker to its transfer peer.""" + raise NotImplementedError + + def finalize(self) -> None: + """Release per-refit backend state.""" + pass + + @abstractmethod + async def send_weights( + self, + weights: Generator[tuple[str, torch.Tensor], None, None], + ) -> None: + """Send ``(name, tensor)`` weights from the policy side.""" + raise NotImplementedError + + @abstractmethod + def receive_weight_batches( + self, + ) -> AsyncGenerator[list[tuple[str, torch.Tensor]], None]: + """Yield ``(name, tensor)`` batches on the generation side.""" + raise NotImplementedError + + +def create_checkpoint_engine( + backend: str, *, bucket_size_bytes: int, engine_kwargs: dict[str, Any] +) -> CheckpointEngine: + if backend == "nixl": + backend = "nemo_rl.utils.checkpoint_engines.nixl:NIXLCheckpointEngine" + if ":" not in backend: + raise ValueError( + f"Unknown checkpoint-engine backend {backend!r}: expected 'nixl' or a " + "'module:ClassName' path to a CheckpointEngine implementation." + ) + module_name, class_name = backend.split(":", 1) + engine_cls = getattr(importlib.import_module(module_name), class_name) + return engine_cls(bucket_size=bucket_size_bytes, **engine_kwargs) + + +def split_weight_chunks( + weights: Generator[tuple[str, torch.Tensor], None, None], bucket_size: int +) -> Generator[tuple[TensorMeta, torch.Tensor], None, None]: + for name, weight in weights: + buffer = weight.contiguous().view(-1).view(torch.uint8) + for chunk_offset in range(0, weight.nbytes, bucket_size): + chunk_size = min(bucket_size, weight.nbytes - chunk_offset) + yield ( + TensorMeta( + name, weight.shape, weight.dtype, chunk_offset, chunk_size, None + ), + buffer[chunk_offset : chunk_offset + chunk_size], + ) + + +async def merge_weight_chunk_batches( + chunk_batches: AsyncGenerator[list[tuple[TensorMeta, torch.Tensor]], None], + *, + merge_device: torch.device | str | None = None, +) -> AsyncGenerator[list[tuple[str, torch.Tensor]], None]: + merge_weight: torch.Tensor | None = None + merge_offset = 0 + async for chunk_batch in chunk_batches: + weight_batch: list[tuple[str, torch.Tensor]] = [] + for meta, chunk in chunk_batch: + if meta.chunk_offset == 0 and meta.chunk_size == meta.nbytes: + weight_batch.append( + (meta.name, chunk.view(meta.dtype).view(meta.shape)) + ) + continue + if merge_weight is None: + merge_weight = torch.empty( + meta.shape, + dtype=meta.dtype, + device=merge_device or chunk.device, + ) + merge_offset = 0 + merge_weight.view(-1).view(torch.uint8)[ + meta.chunk_offset : meta.chunk_offset + meta.chunk_size + ] = chunk + merge_offset += meta.chunk_size + if merge_offset == meta.nbytes: + weight_batch.append((meta.name, merge_weight)) + merge_weight = None + merge_offset = 0 + if weight_batch: + yield weight_batch diff --git a/nemo_rl/utils/checkpoint_engines/nixl.py b/nemo_rl/utils/checkpoint_engines/nixl.py new file mode 100644 index 00000000000..7a3a9a01b54 --- /dev/null +++ b/nemo_rl/utils/checkpoint_engines/nixl.py @@ -0,0 +1,472 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import importlib +import uuid +from collections import defaultdict, deque +from collections.abc import AsyncGenerator, Generator +from typing import Any, cast + +import ray +import torch +import zmq +import zmq.asyncio + +from nemo_rl.utils.checkpoint_engines.base import ( + CheckpointEngine, + TensorMeta, + merge_weight_chunk_batches, + split_weight_chunks, +) + +NixlAgentMetadata = dict[str, Any] +NIXL_DEFAULT_BACKEND_NAME = "UCX" +NIXL_TRANSFER_BUFFER_COUNT = 2 + + +def _source_rank_for_rollout( + rollout_rank: int, + *, + train_world_size: int, + rollout_world_size: int, +) -> int: + if train_world_size < 1: + raise ValueError("train_world_size must be >= 1.") + if rollout_world_size < 1: + raise ValueError("rollout_world_size must be >= 1.") + if rollout_rank < 0 or rollout_rank >= rollout_world_size: + raise ValueError( + f"rollout_rank must be in [0, {rollout_world_size}), got {rollout_rank}." + ) + if train_world_size < rollout_world_size: + raise ValueError( + "NIXL checkpoint-engine refit requires train_world_size >= " + f"rollout_world_size, got {train_world_size} < {rollout_world_size}." + ) + + return rollout_rank + + +def _create_nixl_agent( + agent_name: str, + backend_name: str, + backend_init_params: dict[str, Any] | None = None, +) -> Any: + try: + nixl_api = importlib.import_module("nixl._api") + except ImportError as exc: + raise ImportError("Install NIXL or disable checkpoint-engine refit.") from exc + if backend_name == "UCX" and backend_init_params is None: + return nixl_api.nixl_agent(agent_name) + + agent = nixl_api.nixl_agent(agent_name, nixl_api.nixl_agent_config(backends=[])) + agent.create_backend( + backend_name, + {key: str(value) for key, value in (backend_init_params or {}).items()}, + ) + return agent + + +def resolve_nixl_backend_kwargs( + nixl_kwargs: dict[str, Any], +) -> tuple[str, dict[str, Any] | None]: + """Resolve ``(backend_name, backend_init_params)`` from ``engine_kwargs.nixl``. + + Single source for the NIXL backend-name default so preinit call sites don't + each repeat ``.get("backend_name", NIXL_DEFAULT_BACKEND_NAME)``. + """ + return ( + nixl_kwargs.get("backend_name", NIXL_DEFAULT_BACKEND_NAME), + nixl_kwargs.get("backend_init_params"), + ) + + +def preinit_nixl_agent( + *, + backend_name: str = NIXL_DEFAULT_BACKEND_NAME, + backend_init_params: dict[str, Any] | None = None, +) -> Any: + agent = _create_nixl_agent( + f"preinit-{uuid.uuid4()}", backend_name, backend_init_params + ) + agent.get_agent_metadata() + return agent + + +def _sync_device(device: torch.device) -> None: + if device.type == "cuda": + torch.cuda.synchronize(device) + elif torch.cuda.is_available(): + # Pinned host transfer buffers receive asynchronous device-to-host + # copies (non_blocking=True from CUDA weight tensors). Flush the current + # CUDA stream before the peer RDMA-reads the buffer, otherwise the reader + # can observe stale/partial weights with no error. + torch.cuda.current_stream().synchronize() + + +class NixlAgent: + """Wrap a NIXL agent and its peer-control messaging.""" + + def __init__( + self, + backend_name: str = NIXL_DEFAULT_BACKEND_NAME, + backend_init_params: dict[str, Any] | None = None, + ) -> None: + self.agent_name = str(uuid.uuid4()) + self.agent = _create_nixl_agent( + self.agent_name, backend_name, backend_init_params + ) + self.messages: dict[str, deque[dict[str, Any]]] = defaultdict(deque) + self.notifications: dict[str, deque[bytes]] = defaultdict(deque) + self.zmq_clients: dict[str, zmq.Socket] = {} + self.zmq_client_context = zmq.Context() + self.ip = ray.util.get_node_ip_address().strip("[]") + self.zmq_context = zmq.asyncio.Context() + self.socket = self.zmq_context.socket(zmq.PULL) + self.listen_port = self.socket.bind_to_random_port(f"tcp://{self.ip}") + + def get_agent_metadata(self) -> NixlAgentMetadata: + return { + "agent_name": self.agent_name, + "agent_metadata": self.agent.get_agent_metadata(), + "zmq_ip": self.ip, + "zmq_port": self.listen_port, + } + + def add_remote_agent(self, metadata: NixlAgentMetadata) -> str: + remote = self.agent.add_remote_agent(metadata["agent_metadata"]) + agent_name = remote.decode("utf-8") if isinstance(remote, bytes) else remote + socket = self.zmq_client_context.socket(zmq.PUSH) + socket.connect(f"tcp://{metadata['zmq_ip']}:{metadata['zmq_port']}") + self.zmq_clients[agent_name] = socket + return agent_name + + def remove_remote_agent(self, agent_name: str) -> None: + self.agent.remove_remote_agent(agent_name) + self.zmq_clients.pop(agent_name).close(linger=0) + + def send_message(self, agent_name: str, message: dict[str, Any]) -> None: + self.zmq_clients[agent_name].send_pyobj( + (self.agent_name, message), zmq.DONTWAIT + ) + + async def read_message(self, agent_name: str) -> dict[str, Any]: + while not self.messages[agent_name]: + if callable(progress := getattr(self.agent, "progress", None)): + progress() + try: + remote_agent_name, message = await self.socket.recv_pyobj(zmq.DONTWAIT) + except zmq.Again: + await asyncio.sleep(0) + continue + self.messages[remote_agent_name].append(message) + return self.messages[agent_name].popleft() + + async def wait_notification(self, agent_name: str, notify_key: bytes) -> None: + while True: + pending_notifications = self.notifications[agent_name] + for notification in pending_notifications: + if notification == notify_key: + pending_notifications.remove(notification) + return + + if callable(progress := getattr(self.agent, "progress", None)): + progress() + new_notifications = self.agent.get_new_notifs() + for remote_agent_name, notifications in new_notifications.items(): + if isinstance(remote_agent_name, bytes): + remote_agent_name = remote_agent_name.decode("utf-8") + self.notifications[remote_agent_name].extend(notifications) + await asyncio.sleep(0) + + +class NIXLCheckpointEngine(CheckpointEngine): + """Transfer checkpoint weight buckets through a NIXL backend.""" + + def __init__( + self, + bucket_size: int, + device: str | torch.device = "cuda", + backend_name: str = NIXL_DEFAULT_BACKEND_NAME, + backend_init_params: dict[str, Any] | None = None, + shard_expert_weights: bool = False, + release_after_refit: bool = False, + ) -> None: + if bucket_size < 1: + raise ValueError("NIXL checkpoint-engine bucket_size must be >= 1 byte.") + self.bucket_size = bucket_size + self.shard_expert_weights = shard_expert_weights + self.release_after_refit = release_after_refit + self._target_weight_layout: dict[str, Any] | None = None + self.agent = NixlAgent(backend_name, backend_init_params) + self.prev_agent: str | None = None + self.next_agent: str | None = None + self.buffers: list[torch.Tensor] = [] + self.registration_descs: list[Any] = [] + self.xfer_descs: list[Any] = [] + transfer_device = torch.device(device) + if transfer_device.type == "cuda" and transfer_device.index is None: + transfer_device = torch.device("cuda", torch.cuda.current_device()) + self._transfer_device = transfer_device # pyrefly: ignore[read-only] + self._cupy_buffers: list[Any] = [] + self._cupy_memory_pool: Any | None = None + self._uses_torch_cuda_buffers = False + + def _allocate_transfer_buffer(self) -> torch.Tensor: + device = self._transfer_device + if device.type != "cuda": + return torch.zeros( + self.bucket_size, + dtype=torch.uint8, + device=device, + pin_memory=torch.cuda.is_available(), + ) + + torch.cuda.set_device(device) + try: + cupy = importlib.import_module("cupy") + except ImportError: + self._uses_torch_cuda_buffers = True + return torch.zeros(self.bucket_size, dtype=torch.uint8, device=device) + if self._cupy_memory_pool is None: + self._cupy_memory_pool = cupy.cuda.MemoryPool() + with ( + cupy.cuda.Device(device.index), + cupy.cuda.using_allocator(self._cupy_memory_pool.malloc), + ): + cupy_buffer = cupy.zeros(self.bucket_size, dtype=cupy.uint8) + self._cupy_buffers.append(cupy_buffer) + return torch.as_tensor(cupy_buffer, dtype=torch.uint8, device=device) + + def prepare(self) -> NixlAgentMetadata: + if not self.buffers: + self.buffers = [ + self._allocate_transfer_buffer() + for _ in range(NIXL_TRANSFER_BUFFER_COUNT) + ] + for buffer in self.buffers: + self.registration_descs.append(self.agent.agent.register_memory(buffer)) + self.xfer_descs.append(self.agent.agent.get_xfer_descs(buffer)) + return self.agent.get_agent_metadata() + + def get_target_weight_layout(self) -> dict[str, Any] | None: + return self._target_weight_layout + + def init_policy_process_group( + self, + *, + worker_rank: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[NixlAgentMetadata], + ) -> None: + self._disconnect_peers() + source_to_rollout = { + _source_rank_for_rollout( + rollout_rank, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + ): rollout_rank + for rollout_rank in range(rollout_world_size) + } + rollout_rank = source_to_rollout.get(worker_rank) + if rollout_rank is not None: + target_metadata = metadata[train_world_size + rollout_rank] + if self.shard_expert_weights: + self._target_weight_layout = target_metadata["weight_layout"] + self.next_agent = self.agent.add_remote_agent(target_metadata) + + def init_rollout_process_group( + self, + *, + rollout_rank: int, + train_world_size: int, + rollout_world_size: int, + metadata: list[NixlAgentMetadata], + ) -> None: + self._disconnect_peers() + source_rank = _source_rank_for_rollout( + rollout_rank, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + ) + self.prev_agent = self.agent.add_remote_agent(metadata[source_rank]) + + def _disconnect_peers(self) -> None: + if self.prev_agent is not None: + self.agent.remove_remote_agent(self.prev_agent) + if self.next_agent is not None: + self.agent.remove_remote_agent(self.next_agent) + self.prev_agent = None + self.next_agent = None + self._target_weight_layout = None + + def _release_transfer_buffers(self) -> None: + _sync_device(self._transfer_device) + for registration_desc in self.registration_descs: + self.agent.agent.deregister_memory(registration_desc) + self.registration_descs.clear() + self.xfer_descs.clear() + self.buffers.clear() + self._cupy_buffers.clear() + + if self._cupy_memory_pool is not None: + self._cupy_memory_pool.free_all_blocks() + self._cupy_memory_pool = None + elif self._uses_torch_cuda_buffers: + torch.cuda.empty_cache() + self._uses_torch_cuda_buffers = False + + def finalize(self) -> None: + self._disconnect_peers() + if self.release_after_refit: + self._release_transfer_buffers() + + @torch.no_grad() + async def send_weights( + self, weights: Generator[tuple[str, torch.Tensor], None, None] + ) -> None: + if self.next_agent is None: + # DTensor-backed iterators can run collectives while materializing + # tensors. Ranks without rollout peers must still participate. + for _ in weights: + pass + return + next_agent = cast(str, self.next_agent) + + buffers = self.buffers + descs = self.xfer_descs + buffer_index = 0 + offset: int = 0 + bucket_meta: dict[str, TensorMeta] = {} + pending_key: bytes | None = None + + async def wait_readers(notify_key: bytes | None) -> None: + if notify_key is None: + return + await self.agent.wait_notification(next_agent, notify_key) + + async def flush_bucket( + *, + bucket_buffer_index: int, + bucket_offset: int, + bucket_metadata: dict[str, TensorMeta], + previous_key: bytes | None, + is_last: bool, + ) -> tuple[int, int, dict[str, TensorMeta], bytes | None]: + _sync_device(self._transfer_device) + await wait_readers(previous_key) + notify_key = uuid.uuid4().bytes + metadata = { + "bucket_meta": bucket_metadata, + "notify_key": notify_key, + "is_last": is_last, + "remote_descs": descs[bucket_buffer_index], + } + self.agent.send_message(next_agent, metadata) + next_bucket_meta: dict[str, TensorMeta] = {} + return 1 - bucket_buffer_index, 0, next_bucket_meta, notify_key + + for tensor_meta, chunk in split_weight_chunks(weights, self.bucket_size): + alignment = max(1, tensor_meta.dtype.itemsize) + aligned_offset = ((offset + alignment - 1) // alignment) * alignment + if aligned_offset + tensor_meta.chunk_size > self.bucket_size: + buffer_index, offset, bucket_meta, pending_key = await flush_bucket( + bucket_buffer_index=buffer_index, + bucket_offset=offset, + bucket_metadata=bucket_meta, + previous_key=pending_key, + is_last=False, + ) + aligned_offset = 0 + offset = aligned_offset + tensor_meta.offset = offset + bucket_meta[tensor_meta.name] = tensor_meta + buffers[buffer_index][offset : offset + tensor_meta.chunk_size].copy_( + chunk, non_blocking=True + ) + offset += tensor_meta.chunk_size + + buffer_index, offset, bucket_meta, pending_key = await flush_bucket( + bucket_buffer_index=buffer_index, + bucket_offset=offset, + bucket_metadata=bucket_meta, + previous_key=pending_key, + is_last=True, + ) + await wait_readers(pending_key) + + async def receive_weight_batches( + self, + ) -> AsyncGenerator[list[tuple[str, torch.Tensor]], None]: + # Keep full tensors off GPU when they span multiple RDMA buckets. + async for batch in merge_weight_chunk_batches( + self._receive_weight_chunk_batches(), + merge_device="cpu" if self._transfer_device.type == "cuda" else None, + ): + yield batch + + async def _wait_read(self, xfer_handle: Any, remote_agent: str) -> None: + while True: + if callable(progress := getattr(self.agent.agent, "progress", None)): + progress() + state = self.agent.agent.check_xfer_state(xfer_handle) + if state == "DONE": + self.agent.agent.release_xfer_handle(xfer_handle) + return + if state == "ERR": + raise RuntimeError(f"NIXL read from {remote_agent} failed.") + await asyncio.sleep(0) + + async def _receive_weight_chunk_batches( + self, + ) -> AsyncGenerator[list[tuple[TensorMeta, torch.Tensor]], None]: + prev_agent = self.prev_agent + if prev_agent is None: + raise RuntimeError("NIXL rollout process group is not initialized.") + + buffers = self.buffers + descs = self.xfer_descs + buffer_index = 1 + while True: + message = await self.agent.read_message(prev_agent) + xfer_handle = self.agent.agent.initialize_xfer( + "READ", + descs[buffer_index], + message["remote_descs"], + prev_agent, + message["notify_key"], + ) + if self.agent.agent.transfer(xfer_handle) == "ERR": + raise RuntimeError(f"NIXL read from {prev_agent} failed to start.") + await self._wait_read(xfer_handle, prev_agent) + + chunks = [ + ( + meta, + buffers[buffer_index][ + int(meta.offset) : int(meta.offset) + meta.chunk_size + ], + ) + for meta in message["bucket_meta"].values() + ] + if chunks: + yield chunks + _sync_device(self._transfer_device) + if message["is_last"]: + break + buffer_index = 1 - buffer_index diff --git a/nemo_rl/utils/weight_transfer_sparse_codec.py b/nemo_rl/utils/weight_transfer_sparse_codec.py index 8e8eae54735..52a10f8f3bd 100644 --- a/nemo_rl/utils/weight_transfer_sparse_codec.py +++ b/nemo_rl/utils/weight_transfer_sparse_codec.py @@ -21,7 +21,7 @@ import numpy as np import torch -from nemo_rl.models.generation.vllm.config import VllmRefitConfig +from nemo_rl.models.generation.vllm.config import VllmSparseRefitConfig NamedTensor = tuple[str, torch.Tensor] TensorBatch = list[NamedTensor] @@ -209,7 +209,7 @@ class DeltaCompressionTracker: def __init__( self, - config: VllmRefitConfig, + config: VllmSparseRefitConfig, ) -> None: self.refit_config = config delta_config = config.delta_compression diff --git a/nemo_rl/utils/weight_transfer_stream.py b/nemo_rl/utils/weight_transfer_stream.py index b39d0804df9..d1d74efc8a0 100644 --- a/nemo_rl/utils/weight_transfer_stream.py +++ b/nemo_rl/utils/weight_transfer_stream.py @@ -521,11 +521,12 @@ def stream_sparse_delta_payloads_via_s3_manifest( bucket = (refit_config.storage.s3_bucket or "").strip() if not bucket: raise RuntimeError( - "policy.generation.refit_cfg.storage.s3_bucket must be set for S3 refit." + "policy.generation.refit_cfg.sparse.storage.s3_bucket must be set " + "for S3 refit." ) region = refit_config.storage.s3_region.strip() if not region: - raise ValueError("refit_cfg.storage.s3_region must not be empty.") + raise ValueError("refit_cfg.sparse.storage.s3_region must not be empty.") store = _get_manifest_s3_store(bucket, region) object_prefix = refit_config.storage.s3_prefix.strip("/") run_prefix = ( diff --git a/nemo_rl/weight_sync/checkpoint_engine_config.py b/nemo_rl/weight_sync/checkpoint_engine_config.py new file mode 100644 index 00000000000..8bf7ed3c9cd --- /dev/null +++ b/nemo_rl/weight_sync/checkpoint_engine_config.py @@ -0,0 +1,65 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Any, cast + +from nemo_rl.models.generation.interfaces import ( + CheckpointEngineConfig, + GenerationConfig, +) +from nemo_rl.models.generation.vllm.config import ( + VLLM_SPARSE_REFIT_TRANSPORTS, + VllmCheckpointEnginePluginConfig, + VllmConfig, + normalize_vllm_refit_config, +) + + +def checkpoint_engine_refit_config( + generation_config: GenerationConfig, +) -> CheckpointEngineConfig | None: + """Translate a checkpoint-engine refit scope into the internal schema.""" + config = cast(VllmConfig, generation_config) + transport = config.get("refit_transport") + refit_config = normalize_vllm_refit_config(config) + if ( + refit_config is None + or transport is None + or transport in VLLM_SPARSE_REFIT_TRANSPORTS + ): + return None + + if transport == "nixl": + scoped_config = refit_config.nixl + backend = "nixl" + else: + plugin_config = cast(dict[str, Any], refit_config.model_extra or {}).get( + transport + ) + scoped_config = VllmCheckpointEnginePluginConfig.model_validate(plugin_config) + backend = transport + + engine_kwargs: dict[str, dict[str, Any]] = { + backend: scoped_config.model_dump( + exclude={"update_weights_bucket_memory_ratio"} + ) + } + checkpoint_engine_config: CheckpointEngineConfig = { + "backend": backend, + "update_weights_bucket_memory_ratio": ( + scoped_config.update_weights_bucket_memory_ratio + ), + "engine_kwargs": engine_kwargs, + } + return checkpoint_engine_config diff --git a/nemo_rl/weight_sync/checkpoint_engine_weight_synchronizer.py b/nemo_rl/weight_sync/checkpoint_engine_weight_synchronizer.py new file mode 100644 index 00000000000..9e035fb4ff3 --- /dev/null +++ b/nemo_rl/weight_sync/checkpoint_engine_weight_synchronizer.py @@ -0,0 +1,257 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from contextlib import nullcontext +from dataclasses import dataclass +from typing import Any, Optional + +import ray + +from nemo_rl.models.generation.interfaces import CheckpointEngineConfig +from nemo_rl.utils.timer import Timer +from nemo_rl.weight_sync.interfaces import WeightSynchronizer + +_MEBIBYTE = 1024 * 1024 + + +def _flatten_metadata(results: list[Any]) -> list[Any]: + return [ + item + for result in results + for item in (result if isinstance(result, list) else [result]) + ] + + +def _sort_ranked_metadata(metadata: list[Any]) -> list[Any]: + if all(isinstance(item, dict) and "rank" in item for item in metadata): + return sorted(metadata, key=lambda item: item["rank"]) + return metadata + + +def _ordered_generation_metadata(generation_results: list[Any]) -> list[Any]: + """Order vLLM generation metadata by global rollout rank. + + Each result belongs to one vLLM data-parallel group. Engine-local ranks + are unique only within a group, so sort each group before concatenating + them in worker-group order. + """ + metadata: list[Any] = [] + for group_result in generation_results: + group_metadata = ( + group_result if isinstance(group_result, list) else [group_result] + ) + metadata.extend(_sort_ranked_metadata(group_metadata)) + return metadata + + +@dataclass +class CheckpointEngineWeightSynchronizer(WeightSynchronizer): + """Coordinate checkpoint-engine setup and policy-to-rollout transfers.""" + + _policy: Any + _generation: Any + _checkpoint_engine_config: CheckpointEngineConfig + _stale: bool = True + _checkpoint_engine_ready: bool = False + _bucket_size_bytes: int | None = None + + def init_communicator(self) -> None: + self._generation.prepare_refit_info(self._policy.prepare_refit_info()) + self._ensure_checkpoint_engine_ready() + + @property + def is_stale(self) -> bool: + return self._stale + + def mark_stale(self) -> None: + self._stale = True + + def _release_after_refit(self) -> bool: + cfg = self._checkpoint_engine_config + return bool(cfg["engine_kwargs"][cfg["backend"]]["release_after_refit"]) + + def _run_policy( + self, checkpoint_method: str, **method_kwargs: Any + ) -> list[ray.ObjectRef]: + return self._policy.worker_group.run_all_workers_single_data( + "checkpoint_engine_rpc", + checkpoint_method=checkpoint_method, + method_kwargs=method_kwargs, + ) + + def _generation_rpc(self) -> str: + return ( + "checkpoint_engine_rpc_async" + if self._generation.cfg["vllm_cfg"]["async_engine"] + else "checkpoint_engine_rpc" + ) + + def _run_generation( + self, checkpoint_method: str, method_args: tuple[Any, ...] = () + ) -> list[ray.ObjectRef]: + return self._generation.worker_group.run_all_workers_single_data( + self._generation_rpc(), + checkpoint_method=checkpoint_method, + method_args=method_args, + run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"], + ) + + def _resolve_bucket_size_bytes(self) -> int: + if self._bucket_size_bytes is not None: + return self._bucket_size_bytes + + memory_ratio_raw = self._checkpoint_engine_config[ + "update_weights_bucket_memory_ratio" + ] + try: + memory_ratio = float(memory_ratio_raw) + except (TypeError, ValueError) as exc: + raise ValueError( + "update_weights_bucket_memory_ratio must be a valid float, got " + f"{memory_ratio_raw!r}." + ) from exc + if not 0 < memory_ratio < 1: + raise ValueError( + "update_weights_bucket_memory_ratio must be between 0 and 1, got " + f"{memory_ratio_raw!r}." + ) + + total_memory = _flatten_metadata( + ray.get( + self._run_policy("checkpoint_engine_total_memory_bytes") + + self._run_generation("checkpoint_engine_total_memory_bytes") + ) + ) + minimum_total_bytes = min(int(value) for value in total_memory) + bucket_size_bytes = int(minimum_total_bytes * memory_ratio) + bucket_size_bytes = bucket_size_bytes // _MEBIBYTE * _MEBIBYTE + if bucket_size_bytes < _MEBIBYTE: + raise ValueError( + "Checkpoint-engine bucket sizing produced less than 1 MiB per buffer." + ) + + self._bucket_size_bytes = bucket_size_bytes + print( + "[checkpoint engine] Bucket size: " + f"{bucket_size_bytes // _MEBIBYTE} MiB per buffer " + f"({memory_ratio:.1%} of {minimum_total_bytes / 1024**3:.2f} GiB " + "minimum total GPU memory)." + ) + return bucket_size_bytes + + def _ensure_checkpoint_engine_ready(self) -> None: + if self._checkpoint_engine_ready: + return + + cfg = self._checkpoint_engine_config + backend = cfg["backend"] + bucket_size_bytes = self._resolve_bucket_size_bytes() + engine_kwargs = cfg["engine_kwargs"][backend] + + ray.get( + self._run_policy( + "init_checkpoint_engine", + backend=backend, + bucket_size_bytes=bucket_size_bytes, + engine_kwargs=engine_kwargs, + ) + + self._run_generation( + "init_checkpoint_engine", + (backend, bucket_size_bytes, engine_kwargs), + ) + ) + + policy_prepare_refs = self._run_policy("prepare_checkpoint_engine") + generation_prepare_refs = self._run_generation("prepare_checkpoint_engine") + prepare_results = ray.get(policy_prepare_refs + generation_prepare_refs) + policy_metadata = _sort_ranked_metadata( + _flatten_metadata(prepare_results[: len(policy_prepare_refs)]) + ) + generation_metadata = _ordered_generation_metadata( + prepare_results[len(policy_prepare_refs) :] + ) + topology = { + "metadata": policy_metadata + generation_metadata, + "train_world_size": len(policy_metadata), + "rollout_world_size": len(generation_metadata), + } + worker_count = len(self._generation.worker_group.workers) + workers_per_group = worker_count // self._generation.dp_size + ray.get( + self._run_policy("init_checkpoint_engine_process_group", **topology) + + self._generation.worker_group.run_all_workers_multiple_data( + self._generation_rpc(), + method_args=[ + ( + rank_prefix, + topology["train_world_size"], + topology["rollout_world_size"], + topology["metadata"], + ) + for rank_prefix in range(0, worker_count, workers_per_group) + ], + run_rank_0_only_axes=["tensor_parallel", "pipeline_parallel"], + common_kwargs={ + "checkpoint_method": "init_checkpoint_engine_process_group" + }, + ) + ) + self._checkpoint_engine_ready = True + + def sync_weights( + self, + *, + timer: Optional[Timer] = None, + kv_scales: Optional[dict[str, float]] = None, + ) -> None: + self._stale = True + self._ensure_checkpoint_engine_ready() + context = ( + timer.time("prepare_for_generation/transfer_and_update_weights") + if timer is not None + else nullcontext() + ) + + try: + with context: + policy_refs = self._run_policy( + "send_weights_via_checkpoint_engine", kv_scales=kv_scales + ) + results = ray.get( + policy_refs + + self._run_generation("update_weights_from_checkpoint_engine") + ) + if not all( + result + for result in results[len(policy_refs) :] + if result is not None + ): + raise RuntimeError( + "Weight transfer failed during " + f"{self._checkpoint_engine_config['backend']} " + "checkpoint-engine sync." + ) + self._stale = False + finally: + if self._release_after_refit(): + self.shutdown() + + def shutdown(self) -> None: + if not self._checkpoint_engine_ready: + return + ray.get( + self._run_policy("finalize_checkpoint_engine") + + self._run_generation("finalize_checkpoint_engine") + ) + self._checkpoint_engine_ready = False diff --git a/nemo_rl/weight_sync/factory.py b/nemo_rl/weight_sync/factory.py index 061022fc9c9..505cf955162 100644 --- a/nemo_rl/weight_sync/factory.py +++ b/nemo_rl/weight_sync/factory.py @@ -26,6 +26,9 @@ SGLANG_BACKEND, VLLM_BACKEND, ) +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) from nemo_rl.weight_sync.interfaces import WeightSynchronizer @@ -63,6 +66,30 @@ def create_weight_synchronizer( f"Supported backends: {sorted(_SUPPORTED_BACKENDS)}" ) + checkpoint_engine_config = checkpoint_engine_refit_config(generation.cfg) + if checkpoint_engine_config is not None: + if colocated: + raise ValueError( + "checkpoint-engine refit is only supported for non-colocated " + "generation. Set policy.generation.colocated.enabled=false or " + "set policy.generation.refit_transport=null." + ) + if generation_backend != VLLM_BACKEND: + raise NotImplementedError( + "checkpoint-engine non-colocated refit is only supported for " + f"the vLLM generation backend, got {generation_backend!r}. " + "Support for other generation backends is tracked in " + "https://github.com/NVIDIA-NeMo/RL/issues/3288." + ) + + from nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer import ( + CheckpointEngineWeightSynchronizer, + ) + + return CheckpointEngineWeightSynchronizer( + policy, generation, checkpoint_engine_config + ) + if refit_buffer_size_gb is not None and refit_buffer_size_gb <= 0: raise ValueError("refit_buffer_size_gb must be > 0") diff --git a/nemo_rl/weight_sync/interfaces.py b/nemo_rl/weight_sync/interfaces.py index f0e0817d63a..012540cdc64 100644 --- a/nemo_rl/weight_sync/interfaces.py +++ b/nemo_rl/weight_sync/interfaces.py @@ -18,7 +18,8 @@ logic from both PolicyInterface and GenerationInterface. It owns the transfer of model weights between training and generation components. -Transport-specific implementations (IPC/ZMQ, HTTP, NCCL collectives) each +Transport-specific implementations (IPC/ZMQ, HTTP, NCCL collectives, checkpoint +engines) each encapsulate the transfer lifecycle, so algorithm code never branches on backend type. @@ -52,8 +53,8 @@ class WeightSynchronizer(ABC): Colocated transports (IPC, HTTP) own phase transitions internally (offload_before_refit, prepare_for_generation, offload_after_refit). - The NCCL collective transport is a pure data mover; the orchestrator - handles phases externally. + Non-colocated collective and checkpoint-engine transports are pure data movers; + the orchestrator handles phases externally. """ @abstractmethod @@ -73,8 +74,8 @@ def sync_weights( 5. Restore both sides to their ready state Steps 1-2 and 5 (phase transitions) are only performed by colocated - transports (IPC, HTTP). The NCCL collective transport skips them since - policy and generation run on separate GPUs. + transports (IPC, HTTP). Non-colocated collective and checkpoint-engine + transports skip them since policy and generation run on separate GPUs. Step 4 (verification) is performed explicitly by IPC and NCCL transports, which check ``update_success`` and raise on failure. The diff --git a/nemo_rl/weight_sync/vllm_remote_sparse_weight_synchronizer.py b/nemo_rl/weight_sync/vllm_remote_sparse_weight_synchronizer.py index eda7045d9cd..021afd0429a 100644 --- a/nemo_rl/weight_sync/vllm_remote_sparse_weight_synchronizer.py +++ b/nemo_rl/weight_sync/vllm_remote_sparse_weight_synchronizer.py @@ -59,6 +59,7 @@ def validate_vllm_remote_sparse_refit( vllm_cfg = config["vllm_cfg"] refit_config = normalize_vllm_refit_config(config) assert refit_config is not None + sparse_config = refit_config.sparse if ( colocated or not megatron_enabled @@ -72,10 +73,11 @@ def validate_vllm_remote_sparse_refit( "vLLM, and an unquantized rollout." ) if transport == "vllm_s3_sparse" and not ( - refit_config.storage.s3_bucket and refit_config.storage.s3_bucket.strip() + sparse_config.storage.s3_bucket and sparse_config.storage.s3_bucket.strip() ): raise ValueError( - "vllm_s3_sparse requires policy.generation.refit_cfg.storage.s3_bucket." + "vllm_s3_sparse requires " + "policy.generation.refit_cfg.sparse.storage.s3_bucket." ) return _REMOTE_SPARSE_TRANSPORTS[transport] diff --git a/pyproject.toml b/pyproject.toml index 63b1de062e3..22980e357d3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -65,10 +65,12 @@ dependencies = [ "soundfile>=0.13.1", "nccl4py; sys_platform != 'darwin'", # for non-colocated refit "cuda-bindings; sys_platform != 'darwin'", # for non-colocated refit - "pybase64", # for sglang refit - "awscrt>=0.35.0", # for parallel S3 refit transport - "zstandard", # for sparse refit body compression - "nvidia-cudnn-cu13==9.20.0.48; sys_platform != 'darwin'", # for transformer-engine no build isolation + # Worker venvs run a base sync before backend extras, so NIXL must be available here. + "nixl==1.3.0; sys_platform == 'linux' and (platform_machine == 'x86_64' or platform_machine == 'aarch64')", + "pybase64", # for sglang refit + "awscrt>=0.35.0", # for parallel S3 refit transport + "zstandard", # for sparse refit body compression + "nvidia-cudnn-cu13==9.20.0.48; sys_platform != 'darwin'", # for transformer-engine no build isolation # tilelang — replacement Triton kernel mamba-ssm requires when # Triton >= 3.4.0 on Hopper, see github.com/state-spaces/mamba#640. # Without this, qwen3.5 / nano-v3 / moonlight megatron recipes diff --git a/pyrefly.toml b/pyrefly.toml index 55e57fd32a9..7f7a6ccccb0 100644 --- a/pyrefly.toml +++ b/pyrefly.toml @@ -165,15 +165,20 @@ project-includes = [ "nemo_rl/models/generation/sglang/utils/patches.py", "nemo_rl/models/generation/sglang/utils/ray_utils.py", "nemo_rl/models/generation/vllm/__init__.py", + "nemo_rl/models/generation/vllm/checkpoint_engine.py", + "nemo_rl/models/generation/vllm/collective_rpc.py", "nemo_rl/models/generation/vllm/config.py", "nemo_rl/models/generation/vllm/patches.py", "nemo_rl/models/generation/vllm/quantization/fp8_train_utils.py", "nemo_rl/models/generation/vllm/quantization/mxfp8_utils.py", "nemo_rl/models/generation/vllm/reasoning_parsers/nano_v3_reasoning_parser.py", + "nemo_rl/models/generation/vllm/refit_layout.py", + "nemo_rl/models/generation/vllm/refit_loader.py", "nemo_rl/models/generation/vllm/utils.py", "nemo_rl/models/generation/vllm/vllm_backend.py", "nemo_rl/models/generation/vllm/vllm_sparse_delta.py", "nemo_rl/models/generation/vllm/vllm_sparse_refit.py", + "nemo_rl/models/generation/vllm/worker_utils.py", "nemo_rl/models/huggingface/__init__.py", "nemo_rl/models/megatron/__init__.py", "nemo_rl/models/megatron/draft/__init__.py", @@ -182,11 +187,15 @@ project-includes = [ "nemo_rl/models/policy/utils.py", "nemo_rl/models/policy/workers/__init__.py", "nemo_rl/models/policy/workers/megatron_remote_sparse_refit.py", + "nemo_rl/models/policy/workers/checkpoint_engine.py", "nemo_rl/models/policy/workers/patches.py", "nemo_rl/models/value/__init__.py", "nemo_rl/models/value/config.py", "nemo_rl/models/value/workers/__init__.py", "nemo_rl/utils/__init__.py", + "nemo_rl/utils/checkpoint_engines/__init__.py", + "nemo_rl/utils/checkpoint_engines/base.py", + "nemo_rl/utils/checkpoint_engines/nixl.py", "nemo_rl/utils/checkpoint.py", "nemo_rl/utils/config.py", "nemo_rl/utils/grad_norm.py", @@ -203,6 +212,8 @@ project-includes = [ "nemo_rl/utils/weight_transfer_stream.py", "nemo_rl/utils/weight_transfer_zmq.py", "nemo_rl/weight_sync/__init__.py", + "nemo_rl/weight_sync/checkpoint_engine_config.py", + "nemo_rl/weight_sync/checkpoint_engine_weight_synchronizer.py", "nemo_rl/weight_sync/collective_weight_synchronizer.py", "nemo_rl/weight_sync/factory.py", "nemo_rl/weight_sync/http_weight_synchronizer.py", diff --git a/tests/functional/L1_Functional_Tests_GRPO_2.sh b/tests/functional/L1_Functional_Tests_GRPO_2.sh index 784975a4ce9..97866ad6442 100644 --- a/tests/functional/L1_Functional_Tests_GRPO_2.sh +++ b/tests/functional/L1_Functional_Tests_GRPO_2.sh @@ -38,6 +38,7 @@ run_test fast uv run --no-sync bash ./tests/functional/gdpo_async_grpo.sh run_test fast uv run --no-sync bash ./tests/functional/grpo_fsdp2.sh run_test uv run --no-sync bash ./tests/functional/grpo_multiturn.sh run_test uv run --no-sync bash ./tests/functional/grpo_non_colocated.sh +run_test fast uv run --no-sync bash ./tests/functional/grpo_nixl_non_colocated.sh run_test fast uv run --no-sync bash ./tests/functional/grpo_async_replay_buffer_checkpoint.sh cd ${PROJECT_ROOT}/tests diff --git a/tests/functional/grpo_nixl_non_colocated.sh b/tests/functional/grpo_nixl_non_colocated.sh new file mode 100644 index 00000000000..f27b2029ed1 --- /dev/null +++ b/tests/functional/grpo_nixl_non_colocated.sh @@ -0,0 +1,9 @@ +#!/bin/bash + +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) + +set -eou pipefail + +EXP_NAME=grpo_nixl_non_colocated \ + bash "$SCRIPT_DIR/grpo_non_colocated.sh" \ + policy.generation.refit_transport=nixl diff --git a/tests/functional/grpo_non_colocated.sh b/tests/functional/grpo_non_colocated.sh index 8c65aedda29..1bdb4713a07 100755 --- a/tests/functional/grpo_non_colocated.sh +++ b/tests/functional/grpo_non_colocated.sh @@ -7,7 +7,7 @@ git config --global --add safe.directory $PROJECT_ROOT set -eou pipefail -EXP_NAME=$(basename $0 .sh) +EXP_NAME=${EXP_NAME:-$(basename $0 .sh)} EXP_DIR=$SCRIPT_DIR/$EXP_NAME LOG_DIR=$EXP_DIR/logs JSON_METRICS=$EXP_DIR/metrics.json @@ -41,4 +41,3 @@ uv run tests/json_dump_tb_logs.py $LOG_DIR --output_path $JSON_METRICS uv run tests/check_metrics.py $JSON_METRICS \ 'max(data["train/token_mult_prob_error"]) < 1.05' - diff --git a/tests/unit/algorithms/test_distillation.py b/tests/unit/algorithms/test_distillation.py index c63c4bbf051..bffbd57317d 100644 --- a/tests/unit/algorithms/test_distillation.py +++ b/tests/unit/algorithms/test_distillation.py @@ -862,7 +862,8 @@ def test_noncolocated_inference_requires_explicit_gpus_per_node_single_node(): setup(master_config, tokenizer, dataset, None) -def test_distillation_setup_non_colocated_smoke(monkeypatch): +@pytest.mark.parametrize("refit_transport", [None, "nixl"]) +def test_distillation_setup_non_colocated_smoke(monkeypatch, refit_transport): """Smoke test: calling setup with a non-colocated config should succeed.""" from unittest.mock import MagicMock, patch @@ -877,6 +878,8 @@ def test_distillation_setup_non_colocated_smoke(monkeypatch): "top_p": 1.0, "top_k": None, "backend": "vllm", + "refit_transport": refit_transport, + "refit_cfg": None, "colocated": { "enabled": False, "resources": { @@ -938,6 +941,8 @@ def get_master_address_and_port(self): return ip_port class DummyPolicy: + collective_calls = [] + def __init__(self, *args, **kwargs): pass @@ -948,11 +953,15 @@ def offload_after_refit(self): return None def init_collective(self, *args, **kwargs): + self.collective_calls.append((args, kwargs)) return [MagicMock()] class DummyVllmGeneration: + collective_calls = [] + def __init__(self, *args, **kwargs): - pass + self.cfg = kwargs["config"] + self.weight_synchronizer = None def finish_generation(self): return None @@ -961,6 +970,7 @@ def prepare_refit_info(self, *args, **kwargs): return None def init_collective(self, *args, **kwargs): + self.collective_calls.append((args, kwargs)) return [MagicMock()] with ( @@ -970,6 +980,9 @@ def init_collective(self, *args, **kwargs): patch.object(distil_mod, "StatefulDataLoader"), patch.object(distil_mod, "Policy", DummyPolicy), patch.object(distil_mod, "VllmGeneration", DummyVllmGeneration), + patch.object( + distil_mod, "create_weight_synchronizer" + ) as mock_create_synchronizer, patch.object(distil_mod, "get_nemo_gym_uv_cache_dir") as mock_uv_cache_dir, patch.object(distil_mod, "get_nemo_gym_venv_dir") as mock_uv_venv_dir, patch.object(distil_mod, "ray") as mock_ray, @@ -986,6 +999,15 @@ def init_collective(self, *args, **kwargs): assert result[3] is None mock_uv_cache_dir.assert_not_called() mock_uv_venv_dir.assert_not_called() + if refit_transport == "nixl": + mock_create_synchronizer.assert_called_once() + mock_create_synchronizer.return_value.init_communicator.assert_called_once() + assert not DummyPolicy.collective_calls + assert not DummyVllmGeneration.collective_calls + else: + mock_create_synchronizer.assert_not_called() + assert DummyPolicy.collective_calls + assert DummyVllmGeneration.collective_calls @pytest.mark.parametrize( diff --git a/tests/unit/algorithms/test_grpo_checkpoint_engine.py b/tests/unit/algorithms/test_grpo_checkpoint_engine.py new file mode 100644 index 00000000000..c8ac1630177 --- /dev/null +++ b/tests/unit/algorithms/test_grpo_checkpoint_engine.py @@ -0,0 +1,100 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for GRPO checkpoint-engine refit routing.""" + +from pathlib import Path +from typing import cast +from unittest.mock import MagicMock + +from omegaconf import OmegaConf + +from nemo_rl.utils.config import load_config, register_omegaconf_resolvers + + +def test_nixl_example_is_an_enabled_non_colocated_overlay(): + from nemo_rl.algorithms.grpo import MasterConfig + from nemo_rl.models.generation.vllm.config import ( + VllmConfig, + normalize_vllm_refit_config, + ) + + repo_root = Path(__file__).parents[3] + register_omegaconf_resolvers() + raw_config = load_config( + repo_root / "examples/configs/grpo_math_8B_megatron_nixl.yaml" + ) + resolved_config = OmegaConf.to_container(raw_config, resolve=True) + assert isinstance(resolved_config, dict) + config = MasterConfig(**resolved_config) + + generation = config.policy["generation"] + normalize_vllm_refit_config(cast(VllmConfig, generation)) + assert generation["refit_transport"] == "nixl" + assert generation["refit_cfg"].nixl.update_weights_bucket_memory_ratio == 0.05 + assert not generation["colocated"]["enabled"] + assert config.cluster["num_nodes"] == 2 + + +def test_refit_policy_generation_uses_attached_checkpoint_engine_synchronizer(): + from nemo_rl.algorithms import grpo as grpo_mod + from nemo_rl.models.generation.vllm import VllmGeneration + + policy = object() + kv_scales = {"layer_0": 1.0} + + generation = MagicMock(spec=VllmGeneration) + generation.weight_synchronizer = MagicMock() + generation.weight_synchronizer.sync_weights.return_value = {"transfer_s": 1.0} + + result = grpo_mod.refit_policy_generation( + policy=policy, + policy_generation=generation, + colocated_inference=False, + _refit_buffer_size_gb=2, + timer=None, + kv_scales=kv_scales, + ) + + generation.weight_synchronizer.sync_weights.assert_called_once_with( + timer=None, kv_scales=kv_scales + ) + assert result == {"transfer_s": 1.0} + + +def test_refit_policy_generation_sglang_uses_standard_refit(monkeypatch): + from nemo_rl.algorithms import grpo as grpo_mod + from nemo_rl.models.generation.sglang.sglang_generation import SGLangGeneration + + policy = MagicMock() + policy.stream_weights_via_http.return_value = [object()] + + generation = MagicMock(spec=SGLangGeneration) + generation.get_rollout_engine_urls.return_value = ["http://rollout"] + ray_get = MagicMock() + monkeypatch.setattr(grpo_mod.ray, "get", ray_get) + + grpo_mod.refit_policy_generation( + policy=policy, + policy_generation=generation, + colocated_inference=True, + _refit_buffer_size_gb=2, + ) + + policy.stream_weights_via_http.assert_called_once_with( + rollout_engine_urls=["http://rollout"], + buffer_size_bytes=2 * 1024**3, + ) + ray_get.assert_called_once_with(policy.stream_weights_via_http.return_value) + assert generation.prepare_for_generation.call_count == 2 diff --git a/tests/unit/models/generation/test_vllm_checkpoint_engine.py b/tests/unit/models/generation/test_vllm_checkpoint_engine.py new file mode 100644 index 00000000000..75051899ff1 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_checkpoint_engine.py @@ -0,0 +1,171 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for vLLM checkpoint-engine worker lifecycle helpers.""" + +import asyncio +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest +import torch + + +@pytest.mark.vllm +@pytest.mark.parametrize( + ("rank_prefix", "rank", "group_world_size", "rollout_world_size", "expected"), + [ + (4, 4, 8, 8, 4), + (2, 1, 2, 4, 3), + ], +) +def test_resolve_rollout_rank_handles_external_and_engine_local_dp( + monkeypatch, + rank_prefix, + rank, + group_world_size, + rollout_world_size, + expected, +): + from nemo_rl.models.generation.vllm.checkpoint_engine import resolve_rollout_rank + + monkeypatch.setattr(torch.distributed, "get_rank", lambda: rank) + monkeypatch.setattr(torch.distributed, "get_world_size", lambda: group_world_size) + + assert resolve_rollout_rank(rank_prefix, rollout_world_size) == expected + + +@pytest.mark.vllm +def test_checkpoint_engine_worker_lifecycle(monkeypatch): + from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmCheckpointEngineMixin, + ) + + worker = VllmCheckpointEngineMixin() + worker.checkpoint_engine = MagicMock(shard_expert_weights=False) + worker.checkpoint_engine.prepare.return_value = {"agent": "rollout"} + monkeypatch.setattr(torch.distributed, "get_rank", lambda: 2) + monkeypatch.setattr(torch.distributed, "get_world_size", lambda: 2) + + assert worker.prepare_checkpoint_engine() == {"agent": "rollout", "rank": 2} + worker.init_checkpoint_engine_process_group(4, 3, 2, ["metadata"]) + worker.checkpoint_engine.init_rollout_process_group.assert_called_once_with( + rollout_rank=2, + train_world_size=3, + rollout_world_size=2, + metadata=["metadata"], + ) + + worker.finalize_checkpoint_engine() + worker.checkpoint_engine.finalize.assert_called_once_with() + + +@pytest.mark.vllm +def test_update_weights_from_checkpoint_engine_async_loads_all_batches(monkeypatch): + from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmCheckpointEngineMixin, + ) + + batches = [ + [("a", torch.ones(2, 2))], + [("b", torch.ones(3)), ("c", torch.ones(4))], + ] + + class FakeEngine: + shard_expert_weights = False + + async def receive_weight_batches(self): + for batch in batches: + yield batch + + worker = VllmCheckpointEngineMixin() + worker.checkpoint_engine = FakeEngine() + events = [] + worker._load_weights = lambda batch: events.append( + ("load", [name for name, _weight in batch]) + ) + worker._maybe_process_fp8_kv_cache = lambda: events.append(("fp8",)) + monkeypatch.setattr( + torch.cuda, + "current_stream", + lambda: SimpleNamespace(synchronize=lambda: events.append(("sync",))), + ) + + assert asyncio.run(worker._update_weights_from_checkpoint_engine_async()) is True + assert events == [ + ("load", ["a"]), + ("sync",), + ("load", ["b", "c"]), + ("sync",), + ("fp8",), + ] + + +@pytest.mark.vllm +def test_checkpoint_engine_worker_reports_total_memory(monkeypatch): + from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmCheckpointEngineMixin, + ) + + monkeypatch.setattr(torch.cuda, "current_device", lambda: 2) + get_device_properties = MagicMock(return_value=SimpleNamespace(total_memory=1234)) + monkeypatch.setattr(torch.cuda, "get_device_properties", get_device_properties) + + assert VllmCheckpointEngineMixin().checkpoint_engine_total_memory_bytes() == 1234 + get_device_properties.assert_called_once_with(2) + + +@pytest.mark.vllm +def test_checkpoint_engine_methods_only_exist_on_configured_extension(): + from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmCheckpointEngineMixin, + ) + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtension, + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + assert not issubclass(VllmInternalWorkerExtension, VllmCheckpointEngineMixin) + assert not hasattr(VllmInternalWorkerExtension, "prepare_checkpoint_engine") + assert issubclass( + VllmInternalWorkerExtensionWithCheckpointEngine, + VllmCheckpointEngineMixin, + ) + + +@pytest.mark.vllm +def test_checkpoint_engine_rpc_mixins_reduce_update_results(): + from nemo_rl.models.generation.vllm.checkpoint_engine import ( + VllmAsyncCheckpointEngineRpcMixin, + VllmCheckpointEngineRpcMixin, + ) + + sync_worker = SimpleNamespace( + llm=SimpleNamespace(collective_rpc=MagicMock(return_value=[True, None])) + ) + assert VllmCheckpointEngineRpcMixin.checkpoint_engine_rpc( + sync_worker, "update_weights_from_checkpoint_engine", ("arg",) + ) + sync_worker.llm.collective_rpc.assert_called_once_with( + "update_weights_from_checkpoint_engine", args=("arg",) + ) + + async_worker = SimpleNamespace( + llm=SimpleNamespace(collective_rpc=AsyncMock(return_value=[True, None])) + ) + assert asyncio.run( + VllmAsyncCheckpointEngineRpcMixin.checkpoint_engine_rpc_async( + async_worker, "update_weights_from_checkpoint_engine" + ) + ) diff --git a/tests/unit/models/generation/test_vllm_collective_rpc.py b/tests/unit/models/generation/test_vllm_collective_rpc.py new file mode 100644 index 00000000000..5bdb5d5ae26 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_collective_rpc.py @@ -0,0 +1,63 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for vLLM collective-RPC result resolution.""" + +import asyncio +import concurrent.futures + +from nemo_rl.models.generation.vllm.collective_rpc import resolve_collective_rpc_result + + +def test_resolve_collective_rpc_result_passes_through_plain_values(): + assert asyncio.run(resolve_collective_rpc_result(7)) == 7 + assert asyncio.run(resolve_collective_rpc_result("x")) == "x" + + +def test_resolve_collective_rpc_result_unwraps_awaitables(): + async def make() -> int: + async def inner() -> int: + return 42 + + # An awaitable that itself resolves to another awaitable. + return inner() + + assert asyncio.run(resolve_collective_rpc_result(make())) == 42 + + +def test_resolve_collective_rpc_result_unwraps_concurrent_future(): + async def run(): + future: concurrent.futures.Future = concurrent.futures.Future() + future.set_result("done") + return await resolve_collective_rpc_result(future) + + assert asyncio.run(run()) == "done" + + +def test_resolve_collective_rpc_result_preserves_list_and_tuple_shape(): + async def run(): + async def coro(v): + return v + + future: concurrent.futures.Future = concurrent.futures.Future() + future.set_result("f") + # Nested mix of awaitables, futures, and plain values; list outer, + # tuple inner — shapes must be preserved. + nested = [coro(1), (future, coro(2)), 3] + return await resolve_collective_rpc_result(nested) + + result = asyncio.run(run()) + assert result == [1, ("f", 2), 3] + assert isinstance(result, list) + assert isinstance(result[1], tuple) diff --git a/tests/unit/models/generation/test_vllm_generation.py b/tests/unit/models/generation/test_vllm_generation.py index 3017aa9941a..85b63022900 100644 --- a/tests/unit/models/generation/test_vllm_generation.py +++ b/tests/unit/models/generation/test_vllm_generation.py @@ -459,6 +459,17 @@ def test_configure_generation_config_uses_real_delta_baseline(transport: str): assert configured["vllm_cfg"]["load_format"] == "auto" +def test_configure_generation_config_keeps_dummy_startup_weights_for_nixl(): + vllm_config = deepcopy(basic_vllm_test_config) + vllm_config["refit_transport"] = "nixl" + + configured = configure_generation_config( + vllm_config, MagicMock(pad_token_id=0, eos_token_id=1) + ) + + assert configured["vllm_cfg"]["load_format"] == "dummy" + + def test_configure_generation_config_keeps_dummy_startup_weights_with_draft_refit(): """Speculative training can keep dummy startup weights when draft refit is available.""" vllm_config = deepcopy(basic_vllm_test_config) diff --git a/tests/unit/models/generation/test_vllm_modelopt_real_quant_config.py b/tests/unit/models/generation/test_vllm_modelopt_real_quant_config.py index f1d75458d6b..4e71431790f 100644 --- a/tests/unit/models/generation/test_vllm_modelopt_real_quant_config.py +++ b/tests/unit/models/generation/test_vllm_modelopt_real_quant_config.py @@ -124,16 +124,43 @@ def __init__(self, quant_cfg, **kwargs): ) +def _install_fake_vllm_worker(monkeypatch): + """Install the minimal vLLM worker hierarchy needed by the backend import.""" + module_names = ["vllm", "vllm.v1", "vllm.v1.worker"] + modules = {} + for module_name in module_names: + module = types.ModuleType(module_name) + module.__path__ = [] + modules[module_name] = module + monkeypatch.setitem(sys.modules, module_name, module) + + gpu_worker_module = types.ModuleType("vllm.v1.worker.gpu_worker") + + class FakeVllmWorker: + pass + + gpu_worker_module.Worker = FakeVllmWorker + monkeypatch.setitem(sys.modules, "vllm.v1.worker.gpu_worker", gpu_worker_module) + modules["vllm"].v1 = modules["vllm.v1"] + modules["vllm.v1"].worker = modules["vllm.v1.worker"] + modules["vllm.v1.worker"].gpu_worker = gpu_worker_module + + +def _clear_vllm_backend_modules(monkeypatch): + for module_name in ( + "nemo_rl.modelopt.models.generation.vllm_quant_backend", + "nemo_rl.models.generation.vllm.vllm_backend", + ): + monkeypatch.delitem(sys.modules, module_name, raising=False) + + def _import_vllm_quant_backend(monkeypatch): """Import the NeMo-RL backend without requiring the vLLM C extension.""" monkeypatch.delenv("VLLM_MODELOPT_REAL_QUANT", raising=False) - vllm_module = types.ModuleType("vllm") - vllm_module.__path__ = [] - monkeypatch.setitem(sys.modules, "vllm", vllm_module) + _install_fake_vllm_worker(monkeypatch) _install_fake_vllm_reload(monkeypatch) _install_fake_modelopt_tensor_quantizer(monkeypatch) - sys.modules.pop("nemo_rl.modelopt.models.generation.vllm_quant_backend", None) - sys.modules.pop("nemo_rl.models.generation.vllm.vllm_backend", None) + _clear_vllm_backend_modules(monkeypatch) try: return importlib.import_module( "nemo_rl.modelopt.models.generation.vllm_quant_backend" @@ -728,6 +755,36 @@ def test_quant_worker_forwards_snapshot_pythonpath_to_inner_vllm_workers(): assert "PYTHONPATH" in worker_mod._EXTRA_ENV_VARS +def test_configure_quant_engine_kwargs_preserves_checkpoint_extension(monkeypatch): + worker_mod = pytest.importorskip( + "nemo_rl.modelopt.models.generation.vllm_quant_worker" + ) + monkeypatch.delenv("VLLM_QUANT_CFG", raising=False) + monkeypatch.delenv("VLLM_MODELOPT_REAL_QUANT", raising=False) + cfg = { + "quant_cfg": "examples/modelopt/quant_configs/nvfp4_w4a8_fp8.yaml", + "refit_transport": "nixl", + "refit_cfg": {"nixl": {}}, + } + llm_kwargs = {} + + worker_mod._configure_quant_engine_kwargs(cfg, llm_kwargs) + + assert llm_kwargs["worker_extension_cls"] == ( + "nemo_rl.modelopt.models.generation.vllm_quant_backend." + "VllmQuantInternalWorkerExtensionWithCheckpointEngine" + ) + + +def test_fake_quant_worker_inherits_nixl_worker(): + patch_mod = pytest.importorskip( + "nemo_rl.modelopt.models.generation.vllm_quant_patch" + ) + from nemo_rl.models.generation.vllm.vllm_backend import NixlVllmWorker + + assert issubclass(patch_mod.FakeQuantWorker, NixlVllmWorker) + + def test_configure_quant_engine_kwargs_for_real_quant(monkeypatch): worker_mod = pytest.importorskip( "nemo_rl.modelopt.models.generation.vllm_quant_worker" @@ -975,14 +1032,14 @@ def test_vllm_modelopt_backend_registers_real_quant_configs_on_import(monkeypatc calls = [] monkeypatch.setenv("VLLM_MODELOPT_REAL_QUANT", "1") - monkeypatch.setitem(sys.modules, "vllm", types.ModuleType("vllm")) + _install_fake_vllm_worker(monkeypatch) _install_fake_modelopt_tensor_quantizer(monkeypatch) monkeypatch.setattr( vllm_modelopt, "register_nemo_modelopt_nvfp4", lambda: calls.append("registered"), ) - sys.modules.pop("nemo_rl.modelopt.models.generation.vllm_quant_backend", None) + _clear_vllm_backend_modules(monkeypatch) importlib.import_module("nemo_rl.modelopt.models.generation.vllm_quant_backend") diff --git a/tests/unit/models/generation/test_vllm_nixl_worker.py b/tests/unit/models/generation/test_vllm_nixl_worker.py new file mode 100644 index 00000000000..09bf0ac5d43 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_nixl_worker.py @@ -0,0 +1,145 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for NIXL preinitialization through vLLM's worker class hook.""" + +from types import SimpleNamespace + +import pytest + +from nemo_rl.models.generation.vllm.checkpoint_engine import ( + NIXL_VLLM_WORKER, + configure_nixl_worker, + preinit_nixl_from_vllm_config, +) + + +def _nixl_config() -> dict: + return { + "backend": "nixl", + "update_weights_bucket_memory_ratio": 0.05, + "engine_kwargs": { + "nixl": { + "device": "cuda", + "backend_name": "UCX", + "backend_init_params": {"foo": "bar"}, + "release_after_refit": False, + "shard_expert_weights": False, + } + }, + } + + +@pytest.mark.parametrize( + "generation_config", + [ + {}, + {"refit_transport": "vllm_zmq_sparse"}, + { + "refit_transport": "custom.module:Engine", + "refit_cfg": {"custom.module:Engine": {}}, + }, + ], +) +def test_configure_nixl_worker_ignores_other_configs(generation_config): + vllm_kwargs = {"additional_config": {"existing": True}} + + configure_nixl_worker(generation_config, vllm_kwargs) + + assert vllm_kwargs == {"additional_config": {"existing": True}} + + +def test_configure_nixl_worker_uses_vllm_extension_points(): + checkpoint_config = _nixl_config() + vllm_kwargs = {"additional_config": {"existing": True}} + + configure_nixl_worker( + { + "refit_transport": "nixl", + "refit_cfg": { + "nixl": { + "backend_name": "UCX", + "backend_init_params": {"foo": "bar"}, + } + }, + }, + vllm_kwargs, + ) + + assert vllm_kwargs["worker_cls"] == NIXL_VLLM_WORKER + assert vllm_kwargs["additional_config"] == { + "existing": True, + "nemo_rl_checkpoint_engine": checkpoint_config, + } + + +def test_configure_nixl_worker_rejects_incompatible_worker_class(): + with pytest.raises(ValueError, match="worker_cls to be unset"): + configure_nixl_worker( + {"refit_transport": "nixl"}, + {"worker_cls": "custom.Worker"}, + ) + + +def test_preinit_nixl_from_vllm_config_uses_configured_backend(monkeypatch): + from nemo_rl.utils.checkpoint_engines import nixl + + calls = [] + agent = object() + monkeypatch.setattr( + nixl, + "preinit_nixl_agent", + lambda **kwargs: calls.append(kwargs) or agent, + ) + config = SimpleNamespace( + additional_config={"nemo_rl_checkpoint_engine": _nixl_config()} + ) + + assert preinit_nixl_from_vllm_config(config) is agent + assert calls == [ + { + "backend_name": "UCX", + "backend_init_params": {"foo": "bar"}, + } + ] + + +def test_preinit_nixl_from_vllm_config_is_disabled_without_nixl_config(): + config = SimpleNamespace(additional_config={}) + + assert preinit_nixl_from_vllm_config(config) is None + + +@pytest.mark.vllm +def test_nixl_worker_preinitializes_before_vllm_worker(monkeypatch): + from nemo_rl.models.generation.vllm import vllm_backend + + calls = [] + agent = object() + monkeypatch.setattr( + vllm_backend, + "preinit_nixl_from_vllm_config", + lambda config: calls.append(("preinit", config)) or agent, + ) + base_worker = vllm_backend.NixlVllmWorker.__bases__[0] + monkeypatch.setattr( + base_worker, + "__init__", + lambda self, config, *args, **kwargs: calls.append(("vllm", config)), + ) + config = object() + worker = vllm_backend.NixlVllmWorker(config) + + assert calls == [("preinit", config), ("vllm", config)] + assert worker._nrl_nixl_preinit_agent is agent diff --git a/tests/unit/models/generation/test_vllm_refit_layout.py b/tests/unit/models/generation/test_vllm_refit_layout.py new file mode 100644 index 00000000000..c9e7e23fb25 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_refit_layout.py @@ -0,0 +1,161 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pytest +import torch + +from nemo_rl.models.generation.vllm.refit_layout import ( + parse_hf_expert_weight, + select_hf_weight_for_vllm_target, +) + + +def _layout(*, tp_rank=0, tp_size=1, local_expert_ids=None): + return { + "expert_params": { + "model.layers.0.mlp.experts.w13_weight": { + "tp_rank": tp_rank, + "tp_size": tp_size, + "local_expert_ids": local_expert_ids, + }, + "model.layers.0.mlp.experts.w2_weight": { + "tp_rank": tp_rank, + "tp_size": tp_size, + "local_expert_ids": local_expert_ids, + }, + }, + "missing_weight_prefixes": [], + } + + +def test_parse_hf_expert_weight_maps_vllm_fused_parameters(): + gate = parse_hf_expert_weight("model.layers.0.mlp.experts.7.gate_proj.weight") + down = parse_hf_expert_weight("model.layers.0.mlp.experts.7.down_proj.weight") + + assert gate is not None + assert (gate.parameter_name, gate.expert_id, gate.shard_id, gate.tp_shard_dim) == ( + "model.layers.0.mlp.experts.w13_weight", + 7, + "w1", + 0, + ) + assert down is not None + assert (down.parameter_name, down.shard_id, down.tp_shard_dim) == ( + "model.layers.0.mlp.experts.w2_weight", + "w2", + 1, + ) + + +def test_select_hf_weight_uses_destination_tp_coordinate(): + gate = torch.arange(32).reshape(8, 4) + down = torch.arange(32).reshape(4, 8) + layout = _layout(tp_rank=1, tp_size=2) + + selected_gate = select_hf_weight_for_vllm_target( + "model.layers.0.mlp.experts.0.gate_proj.weight", + gate, + target_layout=layout, + ) + selected_down = select_hf_weight_for_vllm_target( + "model.layers.0.mlp.experts.0.down_proj.weight", + down, + target_layout=layout, + ) + + torch.testing.assert_close(selected_gate, gate[4:]) + torch.testing.assert_close(selected_down, down[:, 4:]) + + +def test_select_hf_weight_filters_ep_ownership_without_tp_slicing(): + local_weight = torch.arange(32).reshape(8, 4) + remote_weight = local_weight + 100 + layout = _layout(local_expert_ids=[1, 3]) + + selected = select_hf_weight_for_vllm_target( + "model.layers.0.mlp.experts.3.up_proj.weight", + local_weight, + target_layout=layout, + ) + skipped = select_hf_weight_for_vllm_target( + "model.layers.0.mlp.experts.2.up_proj.weight", + remote_weight, + target_layout=layout, + ) + + assert selected is local_weight + assert skipped is None + + +def test_select_hf_weight_applies_intra_expert_tp_after_ep_ownership(): + weight = torch.arange(32).reshape(8, 4) + + selected = select_hf_weight_for_vllm_target( + "model.layers.0.mlp.experts.3.gate_proj.weight", + weight, + target_layout=_layout(tp_rank=1, tp_size=2, local_expert_ids=[1, 3]), + ) + + torch.testing.assert_close(selected, weight[4:]) + + +def test_select_hf_weight_rejects_nondivisible_tp_layout(): + with pytest.raises(ValueError, match="across vLLM TP size 3"): + select_hf_weight_for_vllm_target( + "model.layers.0.mlp.experts.3.gate_proj.weight", + torch.ones(8, 4), + target_layout=_layout(tp_size=3, local_expert_ids=[3]), + ) + + +def test_select_hf_weight_leaves_non_expert_weights_unchanged(): + weight = torch.arange(16).reshape(4, 4) + + selected = select_hf_weight_for_vllm_target( + "model.layers.0.self_attn.q_proj.weight", + weight, + target_layout=_layout(local_expert_ids=[]), + ) + + assert selected is weight + + +@pytest.mark.parametrize( + "name", + [ + "model.layers.1.self_attn.q_proj.weight", + "model.layers.1.mlp.experts.0.gate_proj.weight", + ], +) +def test_select_hf_weight_skips_weights_absent_from_destination_stage(name): + layout = _layout(local_expert_ids=[0]) + layout["missing_weight_prefixes"] = ["model.layers.1."] + + selected = select_hf_weight_for_vllm_target( + name, + torch.ones(4, 4), + target_layout=layout, + ) + + assert selected is None + + +def test_select_hf_weight_skips_experts_absent_from_destination_stage(): + selected = select_hf_weight_for_vllm_target( + "model.layers.1.mlp.experts.0.gate_proj.weight", + torch.ones(4, 4), + target_layout=_layout(local_expert_ids=[0]), + ) + + assert selected is None diff --git a/tests/unit/models/generation/test_vllm_refit_loader.py b/tests/unit/models/generation/test_vllm_refit_loader.py new file mode 100644 index 00000000000..999cf124e84 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_refit_loader.py @@ -0,0 +1,512 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for destination-local vLLM refit loading.""" + +from types import SimpleNamespace + +import pytest +import torch + + +class _FakeUnquantizedMethod: + def __init__(self, backend_name="TRITON"): + self.unquantized_backend = SimpleNamespace(name=backend_name) + + +class _FakeExpertOwner: + def __init__( + self, + *, + use_ep, + expert_map=None, + tp_rank=0, + tp_size=1, + backend_name="TRITON", + ): + self.use_ep = use_ep + self._expert_map = expert_map + self.logical_num_experts = 4 + self.global_num_experts = 4 + self.enable_eplb = False + self.tp_rank = tp_rank + self.tp_size = tp_size + self.quant_config = None + self.base_quant_method = _FakeUnquantizedMethod(backend_name) + + def weight_loader(self, *args, **kwargs): + raise AssertionError("The fake weight loader should not be called") + + def _map_global_expert_id_to_local_expert_id(self, expert_id): + if self._expert_map is None: + return expert_id + return int(self._expert_map[expert_id]) + + +def _expert_param(shape, owner): + param = torch.nn.Parameter(torch.zeros(shape), requires_grad=False) + param.weight_loader = owner.weight_loader + return param + + +@pytest.mark.vllm +def test_destination_local_copy_matches_vllm_full_weight_tp_loading(): + from vllm.model_executor.layers.fused_moe.layer import FusedMoE + + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = FusedMoE.__new__(FusedMoE) + torch.nn.Module.__init__(owner) + owner.moe_config = SimpleNamespace(is_act_and_mul=True) + owner.moe_parallel_config = SimpleNamespace(tp_rank=1) + reference_w13 = _expert_param((2, 8, 6), owner) + reference_w2 = _expert_param((2, 6, 4), owner) + destination_w13 = _expert_param((2, 8, 6), owner) + destination_w2 = _expert_param((2, 6, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + full_w1 = torch.arange(64, dtype=torch.float32).reshape(2, 8, 4) + full_w3 = full_w1 + 100 + full_w2 = torch.arange(64, dtype=torch.float32).reshape(2, 4, 8) + 200 + + owner._load_w13( + expert_data=reference_w13.data, + shard_dim=1, + shard_id="w1", + loaded_weight=full_w1, + tp_rank=1, + ) + owner._load_w13( + expert_data=reference_w13.data, + shard_dim=1, + shard_id="w3", + loaded_weight=full_w3, + tp_rank=1, + ) + owner._load_w2( + expert_data=reference_w2.data, + shard_dim=2, + loaded_weight=full_w2, + tp_rank=1, + ) + + ext._load_destination_local_expert_group( + "w13", destination_w13, "w1", list(enumerate(full_w1[:, 4:])) + ) + ext._load_destination_local_expert_group( + "w13", destination_w13, "w3", list(enumerate(full_w3[:, 4:])) + ) + ext._load_destination_local_expert_group( + "w2", destination_w2, "w2", list(enumerate(full_w2[:, :, 4:])) + ) + + torch.testing.assert_close(destination_w13, reference_w13) + torch.testing.assert_close(destination_w2, reference_w2) + + +@pytest.mark.vllm +def test_refit_load_weights_uses_full_weight_path_by_default(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtension, + ) + + loaded = [] + model = SimpleNamespace(load_weights=lambda *, weights: loaded.extend(weights)) + ext = VllmInternalWorkerExtension.__new__(VllmInternalWorkerExtension) + ext.model_runner = SimpleNamespace( + model=model, + vllm_config=SimpleNamespace(model_config=SimpleNamespace(architectures=[])), + ) + weight = torch.ones(2, 2) + + ext._load_weights([("model.weight", weight)]) + + assert loaded == [("model.weight", weight)] + + +@pytest.mark.vllm +def test_refit_load_weights_dispatches_to_sharded_path_when_enabled(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + loaded = [] + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.checkpoint_engine = SimpleNamespace(shard_expert_weights=True) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + load_weights=lambda **_kwargs: pytest.fail("used full loader") + ), + vllm_config=SimpleNamespace(model_config=SimpleNamespace(architectures=[])), + ) + ext._load_sharded_expert_weights = lambda weights: loaded.extend(weights) + weight = torch.ones(2, 2) + + ext._load_weights([("model.weight", weight)]) + + assert loaded == [("model.weight", weight)] + + +@pytest.mark.vllm +def test_checkpoint_refit_worker_falls_back_to_full_loader(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + loaded = [] + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.checkpoint_engine = SimpleNamespace(shard_expert_weights=False) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace(load_weights=lambda *, weights: loaded.extend(weights)), + vllm_config=SimpleNamespace(model_config=SimpleNamespace(architectures=[])), + ) + weight = torch.ones(2, 2) + + ext._load_weights([("model.weight", weight)]) + + assert loaded == [("model.weight", weight)] + + +@pytest.mark.vllm +def test_checkpoint_refit_preserves_nonsharded_fp8_path(monkeypatch): + from nemo_rl.models.generation.vllm.quantization import fp8 + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + loaded = [] + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.checkpoint_engine = SimpleNamespace(shard_expert_weights=False) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + load_weights=lambda **_kwargs: pytest.fail("used full loader") + ), + vllm_config=SimpleNamespace(model_config=SimpleNamespace(architectures=[])), + ) + monkeypatch.setattr(fp8, "is_fp8_model", lambda _config: True) + + def load_fp8_weights(weights, model_runner): + assert model_runner is ext.model_runner + loaded.extend(weights) + + monkeypatch.setattr(fp8, "load_weights", load_fp8_weights) + weight = torch.ones(2, 2) + + ext._load_weights([("model.weight", weight)]) + + assert loaded == [("model.weight", weight)] + + +@pytest.mark.vllm +def test_refit_load_weights_rejects_sharded_fp8_path(monkeypatch): + from nemo_rl.models.generation.vllm.quantization import fp8 + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.checkpoint_engine = SimpleNamespace(shard_expert_weights=True) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + load_weights=lambda **_kwargs: pytest.fail("used full loader") + ), + vllm_config=SimpleNamespace(model_config=SimpleNamespace(architectures=[])), + ) + monkeypatch.setattr(fp8, "is_fp8_model", lambda _config: True) + + with pytest.raises(ValueError, match="not supported for FP8"): + ext._load_weights([("model.weight", torch.ones(2, 2))]) + + +@pytest.mark.vllm +def test_checkpoint_engine_weight_layout_reports_ep_and_pp_ownership(monkeypatch): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = _FakeExpertOwner( + use_ep=True, expert_map=torch.tensor([-1, 0, -1, 1], dtype=torch.int32) + ) + w13 = _expert_param((2, 8, 4), owner) + w2 = _expert_param((2, 4, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + named_parameters=lambda: [ + ("model.layers.0.mlp.experts.w13_weight", w13), + ("model.layers.0.mlp.experts.w2_weight", w2), + ] + ) + ) + monkeypatch.setattr( + "vllm.model_executor.models.utils.get_pp_missing_layer_names", + lambda model: ["model.layers.1."], + ) + + layout = ext._checkpoint_engine_weight_layout() + + assert layout["expert_params"]["model.layers.0.mlp.experts.w13_weight"] == { + "tp_rank": 0, + "tp_size": 1, + "local_expert_ids": [1, 3], + } + assert layout["missing_weight_prefixes"] == ["model.layers.1."] + + +@pytest.mark.vllm +def test_checkpoint_engine_weight_layout_rejects_shuffled_backend(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = _FakeExpertOwner(use_ep=True, backend_name="FLASHINFER_TRTLLM") + param = _expert_param((2, 8, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + named_parameters=lambda: [("model.layers.0.mlp.experts.w13_weight", param)] + ) + ) + with pytest.raises(ValueError, match="canonical unquantized Triton"): + ext._checkpoint_engine_weight_layout() + + +@pytest.mark.vllm +def test_checkpoint_engine_weight_layout_rejects_transposed_experts(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = _FakeExpertOwner(use_ep=True) + param = _expert_param((2, 8, 4), owner) + param.is_transposed = True + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + named_parameters=lambda: [("model.layers.0.mlp.experts.w13_weight", param)] + ) + ) + with pytest.raises(ValueError, match="canonical expert-weight orientation"): + ext._checkpoint_engine_weight_layout() + + +@pytest.mark.vllm +def test_sharded_refit_directly_loads_full_ep_owned_experts(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = _FakeExpertOwner( + use_ep=True, expert_map=torch.tensor([-1, 0, -1, 1], dtype=torch.int32) + ) + w13 = _expert_param((2, 8, 4), owner) + w2 = _expert_param((2, 4, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + named_parameters=lambda: [ + ("model.layers.0.mlp.experts.w13_weight", w13), + ("model.layers.0.mlp.experts.w2_weight", w2), + ] + ) + ) + ext.state_dict_info = { + "model.layers.0.mlp.experts.1.gate_proj.weight": ( + torch.Size([4, 4]), + torch.float32, + ), + "model.layers.0.mlp.experts.3.gate_proj.weight": ( + torch.Size([4, 4]), + torch.float32, + ), + "model.layers.0.mlp.experts.1.up_proj.weight": ( + torch.Size([4, 4]), + torch.float32, + ), + "model.layers.0.mlp.experts.3.up_proj.weight": ( + torch.Size([4, 4]), + torch.float32, + ), + "model.layers.0.mlp.experts.1.down_proj.weight": ( + torch.Size([4, 4]), + torch.float32, + ), + "model.layers.0.mlp.experts.3.down_proj.weight": ( + torch.Size([4, 4]), + torch.float32, + ), + } + weights = { + "model.layers.0.mlp.experts.1.gate_proj.weight": torch.full((4, 4), 1.0), + "model.layers.0.mlp.experts.3.gate_proj.weight": torch.full((4, 4), 3.0), + "model.layers.0.mlp.experts.1.up_proj.weight": torch.full((4, 4), 11.0), + "model.layers.0.mlp.experts.3.up_proj.weight": torch.full((4, 4), 13.0), + "model.layers.0.mlp.experts.1.down_proj.weight": torch.full((4, 4), 21.0), + "model.layers.0.mlp.experts.3.down_proj.weight": torch.full((4, 4), 23.0), + } + + remaining = ext._load_sharded_expert_weight_groups(list(weights.items())) + + assert remaining == [] + torch.testing.assert_close( + w13[0, :4], weights["model.layers.0.mlp.experts.1.gate_proj.weight"] + ) + torch.testing.assert_close( + w13[1, :4], weights["model.layers.0.mlp.experts.3.gate_proj.weight"] + ) + torch.testing.assert_close( + w13[0, 4:], weights["model.layers.0.mlp.experts.1.up_proj.weight"] + ) + torch.testing.assert_close( + w13[1, 4:], weights["model.layers.0.mlp.experts.3.up_proj.weight"] + ) + torch.testing.assert_close( + w2[0], weights["model.layers.0.mlp.experts.1.down_proj.weight"] + ) + torch.testing.assert_close( + w2[1], weights["model.layers.0.mlp.experts.3.down_proj.weight"] + ) + + +@pytest.mark.vllm +def test_sharded_refit_loads_sparse_local_expert_ids_individually(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = _FakeExpertOwner(use_ep=False) + param = _expert_param((3, 8, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + expert_0 = torch.full((4, 4), 1.0) + expert_2 = torch.full((4, 4), 2.0) + + ext._load_destination_local_expert_group( + "model.layers.0.mlp.experts.w13_weight", + param, + "w1", + [(0, expert_0), (2, expert_2)], + ) + + torch.testing.assert_close(param[0, :4], expert_0) + torch.testing.assert_close(param[1], torch.zeros(8, 4)) + torch.testing.assert_close(param[2, :4], expert_2) + + +@pytest.mark.vllm +def test_sharded_refit_keeps_full_tp_weight_for_standard_loader(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + owner = _FakeExpertOwner(use_ep=False, tp_rank=0, tp_size=2) + param = _expert_param((4, 8, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext.model_runner = SimpleNamespace( + model=SimpleNamespace( + named_parameters=lambda: [("model.layers.0.mlp.experts.w13_weight", param)] + ) + ) + name = "model.layers.0.mlp.experts.0.gate_proj.weight" + ext.state_dict_info = {name: (torch.Size([8, 4]), torch.float32)} + full_weight = torch.ones(8, 4) + + remaining = ext._load_sharded_expert_weight_groups([(name, full_weight)]) + + assert len(remaining) == 1 + assert remaining[0][0] == name + assert remaining[0][1] is full_weight + torch.testing.assert_close(param, torch.zeros_like(param)) + + +@pytest.mark.vllm +def test_sharded_refit_requires_bound_vllm_expert_loader(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + param_name = "model.layers.0.mlp.experts.w13_weight" + weight_name = "model.layers.0.mlp.experts.0.gate_proj.weight" + param = torch.nn.Parameter(torch.zeros(1, 8, 4), requires_grad=False) + param.weight_loader = lambda *args, **kwargs: None + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext._nrl_named_parameters = {param_name: param} + ext.state_dict_info = {weight_name: (torch.Size([8, 4]), torch.float32)} + + with pytest.raises(RuntimeError, match="Could not resolve.*w13_weight"): + ext._load_sharded_expert_weight_groups([(weight_name, torch.zeros(4, 4))]) + + +@pytest.mark.vllm +def test_sharded_refit_rejects_noncanonical_expert_dimensions(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + param_name = "model.layers.0.mlp.experts.w13_weight" + weight_name = "model.layers.0.mlp.experts.0.gate_proj.weight" + owner = _FakeExpertOwner(use_ep=False) + param = _expert_param((8, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext._nrl_named_parameters = {param_name: param} + ext.state_dict_info = {weight_name: (torch.Size([8, 4]), torch.float32)} + + with pytest.raises(ValueError, match="requires a 3-D vLLM parameter"): + ext._load_sharded_expert_weight_groups([(weight_name, torch.zeros(4, 4))]) + + +@pytest.mark.vllm +def test_sharded_refit_validates_source_shape_against_reported_tp_size(): + from nemo_rl.models.generation.vllm.vllm_backend import ( + VllmInternalWorkerExtensionWithCheckpointEngine, + ) + + param_name = "model.layers.0.mlp.experts.w13_weight" + weight_name = "model.layers.0.mlp.experts.0.gate_proj.weight" + owner = _FakeExpertOwner(use_ep=False, tp_size=2) + param = _expert_param((1, 8, 4), owner) + ext = VllmInternalWorkerExtensionWithCheckpointEngine.__new__( + VllmInternalWorkerExtensionWithCheckpointEngine + ) + ext._nrl_named_parameters = {param_name: param} + ext.state_dict_info = {weight_name: (torch.Size([8, 4]), torch.float32)} + + with pytest.raises(ValueError, match=r"expected \(4, 4\).*TP size 2"): + ext._load_sharded_expert_weight_groups([(weight_name, torch.zeros(2, 4))]) diff --git a/tests/unit/models/generation/test_vllm_worker_helpers.py b/tests/unit/models/generation/test_vllm_worker_helpers.py new file mode 100644 index 00000000000..29d61934447 --- /dev/null +++ b/tests/unit/models/generation/test_vllm_worker_helpers.py @@ -0,0 +1,51 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for vLLM worker helper functions.""" + +import pytest + +from nemo_rl.models.generation.vllm.worker_utils import ( + resolve_data_parallel_local_rank, + resolve_distributed_executor_backend, +) + + +@pytest.mark.parametrize( + ("tp", "pp", "ep", "expected"), + [ + (2, 1, 2, "ray"), + (1, 2, 2, "ray"), + (1, 1, 8, "uni"), + (1, 1, 1, None), + ], +) +def test_resolve_distributed_executor_backend(tp, pp, ep, expected): + assert resolve_distributed_executor_backend(tp, pp, ep) == expected + + +@pytest.mark.parametrize( + ("rank", "model_parallel_size", "executor_backend", "expected"), + [ + (7, 1, "uni", 0), + (6, 2, "ray", 3), + ], +) +def test_resolve_data_parallel_local_rank( + rank, model_parallel_size, executor_backend, expected +): + assert ( + resolve_data_parallel_local_rank(rank, model_parallel_size, executor_backend) + == expected + ) diff --git a/tests/unit/models/policy/test_dtensor_checkpoint_engine.py b/tests/unit/models/policy/test_dtensor_checkpoint_engine.py new file mode 100644 index 00000000000..de77a6081c3 --- /dev/null +++ b/tests/unit/models/policy/test_dtensor_checkpoint_engine.py @@ -0,0 +1,80 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for DTensor checkpoint-engine weight transfer.""" + +import pytest +import torch + + +def test_dtensor_checkpoint_engine_weight_iterator(): + from nemo_rl.models.policy.workers.dtensor_policy_worker import ( + DTensorPolicyWorkerImpl, + ) + + worker = object.__new__(DTensorPolicyWorkerImpl) + worker.model = torch.nn.Linear(2, 1) + worker.dtype = torch.float32 + + weights = list(DTensorPolicyWorkerImpl._checkpoint_engine_weight_iterator(worker)) + + assert [name for name, _tensor in weights] == ["weight", "bias"] + for _name, tensor in weights: + assert tensor.dtype == torch.float32 + assert tensor.is_contiguous() + + +def test_dtensor_checkpoint_engine_rejects_kv_scales(): + from nemo_rl.models.policy.workers.dtensor_policy_worker import ( + DTensorPolicyWorkerImpl, + ) + + worker = object.__new__(DTensorPolicyWorkerImpl) + worker.model = torch.nn.Linear(2, 1) + worker.dtype = torch.float32 + + with pytest.raises(NotImplementedError, match="FP8 kvcache"): + DTensorPolicyWorkerImpl._checkpoint_engine_weight_iterator( + worker, kv_scales={"scale": 1.0} + ) + + +def test_dtensor_checkpoint_engine_cpu_offload_hooks(): + from nemo_rl.models.policy.workers.dtensor_policy_worker import ( + DTensorPolicyWorkerImpl, + ) + + worker = object.__new__(DTensorPolicyWorkerImpl) + worker.model = "cpu_model" + worker.cpu_offload = True + calls = [] + + def move_to_cuda(model): + calls.append(("cuda", model)) + return "cuda_model" + + def move_to_cpu(model): + calls.append(("cpu", model)) + return "cpu_model" + + worker.move_to_cuda = move_to_cuda + worker.move_to_cpu = move_to_cpu + + with pytest.warns(UserWarning, match="cpu_offload adds an onload/offload cycle"): + DTensorPolicyWorkerImpl._prepare_checkpoint_engine_weight_send(worker) + assert worker.model == "cuda_model" + DTensorPolicyWorkerImpl._finalize_checkpoint_engine_weight_send(worker) + + assert worker.model == "cpu_model" + assert calls == [("cuda", "cpu_model"), ("cpu", "cuda_model")] diff --git a/tests/unit/models/policy/test_dtensor_v2_checkpoint_engine.py b/tests/unit/models/policy/test_dtensor_v2_checkpoint_engine.py new file mode 100644 index 00000000000..9d7d5d9e5c3 --- /dev/null +++ b/tests/unit/models/policy/test_dtensor_v2_checkpoint_engine.py @@ -0,0 +1,84 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for Automodel DTensor checkpoint-engine weight transfer.""" + +import pytest +import torch +import torch.nn as nn + +try: + from nemo_rl.models.policy.workers.dtensor_policy_worker_v2 import ( + DTensorPolicyWorkerV2Impl, + ) + + NEMO_AUTOMODEL_AVAILABLE = True +except ImportError: + NEMO_AUTOMODEL_AVAILABLE = False + + +@pytest.mark.automodel +@pytest.mark.skipif(not NEMO_AUTOMODEL_AVAILABLE, reason="nemo_automodel not available") +def test_dtensor_v2_checkpoint_engine_weight_iterator(): + worker = object.__new__(DTensorPolicyWorkerV2Impl) + worker.model = nn.Linear(2, 1) + worker.dtype = torch.float32 + + weights = list(DTensorPolicyWorkerV2Impl._checkpoint_engine_weight_iterator(worker)) + + assert [name for name, _tensor in weights] == ["weight", "bias"] + for _name, tensor in weights: + assert tensor.dtype == torch.float32 + assert tensor.is_contiguous() + + +@pytest.mark.automodel +@pytest.mark.skipif(not NEMO_AUTOMODEL_AVAILABLE, reason="nemo_automodel not available") +def test_dtensor_v2_checkpoint_engine_rejects_kv_scales(): + worker = object.__new__(DTensorPolicyWorkerV2Impl) + worker.model = nn.Linear(2, 1) + worker.dtype = torch.float32 + + with pytest.raises(NotImplementedError, match="FP8 kvcache"): + DTensorPolicyWorkerV2Impl._checkpoint_engine_weight_iterator( + worker, kv_scales={"scale": 1.0} + ) + + +@pytest.mark.automodel +@pytest.mark.skipif(not NEMO_AUTOMODEL_AVAILABLE, reason="nemo_automodel not available") +def test_dtensor_v2_checkpoint_engine_cpu_offload_hooks(): + worker = object.__new__(DTensorPolicyWorkerV2Impl) + worker.model = "cpu_model" + worker.cpu_offload = True + calls = [] + + def move_to_cuda(model): + calls.append(("cuda", model)) + return "cuda_model" + + def move_to_cpu(model): + calls.append(("cpu", model)) + return "cpu_model" + + worker.move_to_cuda = move_to_cuda + worker.move_to_cpu = move_to_cpu + + with pytest.warns(UserWarning, match="cpu_offload adds an onload/offload cycle"): + DTensorPolicyWorkerV2Impl._prepare_checkpoint_engine_weight_send(worker) + assert worker.model == "cuda_model" + DTensorPolicyWorkerV2Impl._finalize_checkpoint_engine_weight_send(worker) + + assert worker.model == "cpu_model" + assert calls == [("cuda", "cpu_model"), ("cpu", "cuda_model")] diff --git a/tests/unit/models/policy/test_megatron_checkpoint_engine.py b/tests/unit/models/policy/test_megatron_checkpoint_engine.py new file mode 100644 index 00000000000..6fa05f47f3e --- /dev/null +++ b/tests/unit/models/policy/test_megatron_checkpoint_engine.py @@ -0,0 +1,74 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for Megatron checkpoint-engine weight transfer.""" + +from types import SimpleNamespace + +import torch + +from nemo_rl.models.policy.workers.checkpoint_engine import ( + MegatronCheckpointEngineSendMixin, +) + + +class _Worker(MegatronCheckpointEngineSendMixin): + pass + + +def test_checkpoint_engine_weight_iterator_keeps_weights_without_target_layout(): + worker = _Worker() + weights = [("weight", torch.ones(2))] + worker._iter_params_with_optional_kv_scales = lambda kv_scales=None: iter(weights) + worker.checkpoint_engine = SimpleNamespace(get_target_weight_layout=lambda: None) + + selected = list(worker._checkpoint_engine_weight_iterator()) + + assert selected == weights + + +def test_checkpoint_engine_weight_iterator_filters_for_vllm_layout(): + worker = _Worker() + local_expert = torch.arange(16).reshape(4, 4) + remote_expert = local_expert + 100 + dense_weight = torch.ones(4, 4) + weights = [ + ("model.layers.0.mlp.experts.0.gate_proj.weight", remote_expert), + ("model.layers.0.mlp.experts.1.gate_proj.weight", local_expert), + ("model.layers.0.self_attn.q_proj.weight", dense_weight), + ("model.layers.1.self_attn.q_proj.weight", dense_weight), + ] + worker._iter_params_with_optional_kv_scales = lambda kv_scales=None: iter(weights) + target_layout = { + "expert_params": { + "model.layers.0.mlp.experts.w13_weight": { + "tp_rank": 0, + "tp_size": 1, + "local_expert_ids": [1], + } + }, + "missing_weight_prefixes": ["model.layers.1."], + } + worker.checkpoint_engine = SimpleNamespace( + get_target_weight_layout=lambda: target_layout, + ) + + selected = list(worker._checkpoint_engine_weight_iterator()) + + assert [name for name, _tensor in selected] == [ + "model.layers.0.mlp.experts.1.gate_proj.weight", + "model.layers.0.self_attn.q_proj.weight", + ] + assert selected[0][1] is local_expert + assert selected[1][1] is dense_weight diff --git a/tests/unit/models/policy/test_megatron_remote_sparse_refit.py b/tests/unit/models/policy/test_megatron_remote_sparse_refit.py index 8b1b697cb9b..4e0ce51893e 100644 --- a/tests/unit/models/policy/test_megatron_remote_sparse_refit.py +++ b/tests/unit/models/policy/test_megatron_remote_sparse_refit.py @@ -16,12 +16,12 @@ import torch -from nemo_rl.models.generation.vllm.config import VllmRefitConfig +from nemo_rl.models.generation.vllm.config import VllmSparseRefitConfig from nemo_rl.models.policy.workers.megatron_remote_sparse_refit import ( MegatronRemoteSparseRefit, ) -_REFIT_CONFIG = VllmRefitConfig( +_REFIT_CONFIG = VllmSparseRefitConfig( delta_compression={ "encoding": "overwrite", "sparse_bucket_size_bytes": 1024, @@ -57,7 +57,7 @@ def test_remote_sparse_initializes_canonical_hf_baseline() -> None: def test_remote_sparse_preserves_xor_config() -> None: remote_refit = MegatronRemoteSparseRefit( _worker(), - VllmRefitConfig( + VllmSparseRefitConfig( delta_compression={ "encoding": "xor", "sparse_bucket_size_bytes": 1024, diff --git a/tests/unit/reference_configs/grpo_math_1B.yaml b/tests/unit/reference_configs/grpo_math_1B.yaml index 39aa2459723..990230f9274 100644 --- a/tests/unit/reference_configs/grpo_math_1B.yaml +++ b/tests/unit/reference_configs/grpo_math_1B.yaml @@ -332,8 +332,17 @@ policy: top_k: null stop_token_ids: null stop_strings: null - refit_transport: null # Set to "vllm_s3_sparse" or "vllm_zmq_sparse" for remote sparse-delta refit. - refit_cfg: null # Optional tuning and storage settings for remote sparse-delta refit. + # null = topology default (IPC colocated, NCCL non-colocated). + # Non-colocated vLLM also supports vllm_s3_sparse, vllm_zmq_sparse, and nixl. + refit_transport: null + refit_cfg: + nixl: + update_weights_bucket_memory_ratio: 0.05 + device: cuda + backend_name: UCX + backend_init_params: null + release_after_refit: false + shard_expert_weights: false mcore_generation_config: async_engine: false max_model_len: ${policy.max_total_sequence_length} diff --git a/tests/unit/tools/test_nixl_elastic_rollout_demo.py b/tests/unit/tools/test_nixl_elastic_rollout_demo.py new file mode 100644 index 00000000000..48fe0d29a73 --- /dev/null +++ b/tests/unit/tools/test_nixl_elastic_rollout_demo.py @@ -0,0 +1,100 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from argparse import Namespace + +import pytest +import torch + +from tools import nixl_elastic_rollout_demo as demo + + +def test_parse_sequence_accepts_comma_separated_counts(): + assert demo._parse_sequence("1, 3,2,,4") == [1, 3, 2, 4] + + +@pytest.mark.parametrize("value", ["", "0", "1,-1"]) +def test_parse_sequence_rejects_empty_or_non_positive_counts(value): + with pytest.raises(ValueError): + demo._parse_sequence(value) + + +def test_phase_delta_adds_and_removes_ranks_for_contiguous_target(): + assert demo._phase_delta(active_ranks=[0, 2], target_count=3) == ([1], []) + assert demo._phase_delta(active_ranks=[0, 1, 2, 3], target_count=2) == ( + [], + [2, 3], + ) + + +def test_build_engine_kwargs_uses_current_checkpoint_engine_schema(): + args = Namespace( + device="cuda", + nixl_backend_name="UCX", + ucx_error_handling_mode="peer", + ) + + assert demo._build_engine_kwargs(args) == { + "device": "cuda", + "backend_name": "UCX", + "backend_init_params": {"ucx_error_handling_mode": "peer"}, + "release_after_refit": True, + } + + +def test_tensor_summary_records_shape_dtype_and_sum(): + weights = [ + ("a", torch.tensor([[1, 2], [3, 4]], dtype=torch.int32)), + ("b", torch.tensor([5, 6], dtype=torch.int64)), + ] + + assert demo._tensor_summary(weights) == { + "a": {"shape": [2, 2], "dtype": "torch.int32", "sum": 10}, + "b": {"shape": [2], "dtype": "torch.int64", "sum": 11}, + } + + +def test_validate_results_accepts_rank_matched_results(): + expected = {1: {"weight": {"shape": [1], "dtype": "torch.int32", "sum": 7}}} + + demo._validate_results( + expected_by_rank=expected, + rollout_results=[{"rank": 1, "received": expected[1]}], + ) + + +def test_validate_results_rejects_unexpected_rank(): + with pytest.raises(RuntimeError, match="unexpected rollout rank 2"): + demo._validate_results(expected_by_rank={}, rollout_results=[{"rank": 2}]) + + +def test_validate_results_rejects_mismatch(): + expected = {0: {"weight": {"shape": [1], "dtype": "torch.int32", "sum": 7}}} + + with pytest.raises(RuntimeError, match="Rollout result mismatch"): + demo._validate_results( + expected_by_rank=expected, + rollout_results=[ + { + "rank": 0, + "received": { + "weight": { + "shape": [1], + "dtype": "torch.int32", + "sum": 8, + } + }, + } + ], + ) diff --git a/tests/unit/utils/test_checkpoint_engine.py b/tests/unit/utils/test_checkpoint_engine.py new file mode 100644 index 00000000000..aacd0bb055b --- /dev/null +++ b/tests/unit/utils/test_checkpoint_engine.py @@ -0,0 +1,1329 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for checkpoint-engine primitives and policy-worker integration.""" + +import asyncio +import multiprocessing +import sys +import traceback +from collections import defaultdict, deque +from types import ModuleType, SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest +import torch + +from nemo_rl.models.policy.workers.base_policy_worker import AbstractPolicyWorker +from nemo_rl.models.policy.workers.checkpoint_engine import ( + PolicyCheckpointEngineMixin, + maybe_preinit_nixl_checkpoint_engine, +) +from nemo_rl.utils.checkpoint_engines import nixl as nixl_mod +from nemo_rl.utils.checkpoint_engines.base import ( + CheckpointEngine, + TensorMeta, + create_checkpoint_engine, + merge_weight_chunk_batches, + split_weight_chunks, +) +from nemo_rl.utils.checkpoint_engines.nixl import NIXLCheckpointEngine + + +class _PluginCheckpointEngine(CheckpointEngine): + def __init__(self, bucket_size: int, marker: str) -> None: + self.bucket_size, self.marker = bucket_size, marker + + def prepare(self): + return {"marker": self.marker} + + def init_policy_process_group( + self, + *, + worker_rank, + train_world_size, + rollout_world_size, + metadata, + ): + pass + + def init_rollout_process_group( + self, + *, + rollout_rank, + train_world_size, + rollout_world_size, + metadata, + ): + pass + + async def send_weights(self, weights): + pass + + async def receive_weight_batches(self): + pass + + +class _RecordingCheckpointEngine(CheckpointEngine): + def __init__(self, bucket_size: int) -> None: + self.bucket_size = bucket_size + self.policy_process_group = None + self.sent_weights = None + self.finalized = False + + def prepare(self): + return {"bucket_size": self.bucket_size} + + def init_policy_process_group( + self, + *, + worker_rank, + train_world_size, + rollout_world_size, + metadata, + ): + self.policy_process_group = { + "worker_rank": worker_rank, + "train_world_size": train_world_size, + "rollout_world_size": rollout_world_size, + "metadata": metadata, + } + + def init_rollout_process_group( + self, + *, + rollout_rank, + train_world_size, + rollout_world_size, + metadata, + ): + pass + + def finalize(self) -> None: + self.finalized = True + + async def send_weights(self, weights): + self.sent_weights = list(weights) + + async def receive_weight_batches(self): + pass + + +class _CheckpointPolicyWorker(PolicyCheckpointEngineMixin, AbstractPolicyWorker): + def __init__(self) -> None: + self.rank = 3 + self.events = [] + self.kv_scales = None + + def _checkpoint_engine_weight_iterator(self, kv_scales=None): + self.kv_scales = kv_scales + yield "weight", torch.tensor([1.0, 2.0]) + + def _prepare_checkpoint_engine_weight_send(self) -> None: + self.events.append("prepare") + + def _finalize_checkpoint_engine_weight_send(self) -> None: + self.events.append("finalize") + + +def _run_checkpoint_rpc( + worker: _CheckpointPolicyWorker, + checkpoint_method: str, + method_kwargs: dict | None = None, +): + return asyncio.run( + worker.checkpoint_engine_rpc( + checkpoint_method, + method_kwargs=method_kwargs, + ) + ) + + +class _FakeNixlTransport: + def __init__(self) -> None: + self.agents = {} + + def create_agent(self, agent_name: str): + agent = _FakeNixlAgent(agent_name, self) + self.agents[agent_name] = agent + return agent + + +class _FakeNixlBackend: + def __init__(self, owner, transport: _FakeNixlTransport) -> None: + self.owner = owner + self.transport = transport + self.initialize_calls = [] + self.released_handles = [] + + def register_memory(self, buffer): + return buffer + + def deregister_memory(self, _registration): + return None + + def get_xfer_descs(self, buffer): + return buffer + + def initialize_xfer( + self, + operation, + local_descs, + remote_descs, + remote_agent, + notify_key, + ): + handle = (local_descs, remote_descs, remote_agent, notify_key) + self.initialize_calls.append((operation, *handle)) + return handle + + def transfer(self, handle): + local_descs, remote_descs, remote_agent, notify_key = handle + local_descs.copy_(remote_descs) + self.transport.agents[remote_agent].notifications[self.owner.agent_name].append( + notify_key + ) + return "OK" + + def check_xfer_state(self, _handle): + return "DONE" + + def release_xfer_handle(self, handle): + self.released_handles.append(handle) + + +class _FakeNixlAgent: + def __init__(self, agent_name: str, transport: _FakeNixlTransport) -> None: + self.agent_name = agent_name + self.transport = transport + self.messages = defaultdict(deque) + self.notifications = defaultdict(deque) + self.agent = _FakeNixlBackend(self, transport) + + def get_agent_metadata(self): + return {"agent_name": self.agent_name} + + def add_remote_agent(self, metadata): + return metadata["agent_name"] + + def remove_remote_agent(self, _agent_name): + return None + + def send_message(self, agent_name, message): + self.transport.agents[agent_name].messages[self.agent_name].append(message) + + async def read_message(self, agent_name): + while not self.messages[agent_name]: + await asyncio.sleep(0) + return self.messages[agent_name].popleft() + + async def wait_notification(self, agent_name, notify_key): + while notify_key not in self.notifications[agent_name]: + await asyncio.sleep(0) + self.notifications[agent_name].remove(notify_key) + + +def _fake_nixl_pair(monkeypatch, bucket_size: int, device: str = "cpu"): + transport = _FakeNixlTransport() + agent_names = iter(("policy", "rollout")) + monkeypatch.setattr( + nixl_mod, + "NixlAgent", + lambda *_args, **_kwargs: transport.create_agent(next(agent_names)), + ) + sender = NIXLCheckpointEngine(bucket_size=bucket_size, device=device) + receiver = NIXLCheckpointEngine(bucket_size=bucket_size, device=device) + sender.prepare() + receiver.prepare() + sender.next_agent = "rollout" + receiver.prev_agent = "policy" + return sender, receiver + + +def _nixl_roundtrip_weights(bucket_size: int) -> list[tuple[str, torch.Tensor]]: + spanning_numel = 3 * (bucket_size // torch.float32.itemsize) + 5 + return [ + ("small", torch.arange(13, dtype=torch.bfloat16)), + ( + "spanning", + torch.arange(spanning_numel, dtype=torch.float32).reshape(-1, 1), + ), + ("tail", torch.arange(7, dtype=torch.int64)), + ] + + +def _run_nixl_roundtrip_process( + role: str, + transfer_device: str, + bucket_size: int, + metadata_queue, + peer_metadata_queue, + result_queue, +) -> None: + try: + torch.cuda.set_device(0) + engine = NIXLCheckpointEngine( + bucket_size=bucket_size, + device=transfer_device, + ) + metadata_queue.put((role, engine.prepare())) + metadata = peer_metadata_queue.get(timeout=30) + + if role == "policy": + engine.init_policy_process_group( + worker_rank=0, + train_world_size=1, + rollout_world_size=1, + metadata=metadata, + ) + weights = [ + (name, tensor.cuda()) + for name, tensor in _nixl_roundtrip_weights(bucket_size) + ] + asyncio.run(engine.send_weights(iter(weights))) + details = {} + else: + engine.init_rollout_process_group( + rollout_rank=0, + train_world_size=1, + rollout_world_size=1, + metadata=metadata, + ) + + async def receive_weights(): + received = {} + async for batch in engine.receive_weight_batches(): + received.update((name, tensor.clone()) for name, tensor in batch) + return received + + received = asyncio.run(receive_weights()) + expected = dict(_nixl_roundtrip_weights(bucket_size)) + assert received.keys() == expected.keys() + for name, tensor in received.items(): + torch.testing.assert_close(tensor.cuda(), expected[name].cuda()) + details = { + "buffer_devices": [buffer.device.type for buffer in engine.buffers], + "cupy_buffer_count": len(engine._cupy_buffers), + "received_devices": { + name: tensor.device.type for name, tensor in received.items() + }, + } + + result_queue.put((role, "success", details)) + except BaseException: + result_queue.put((role, "error", traceback.format_exc())) + raise + + +def _run_nixl_subprocess_roundtrip( + transfer_device: str, bucket_size: int +) -> dict[str, object]: + context = multiprocessing.get_context("spawn") + metadata_queue = context.Queue() + peer_metadata_queues = {role: context.Queue() for role in ("policy", "rollout")} + result_queue = context.Queue() + processes = [ + context.Process( + target=_run_nixl_roundtrip_process, + args=( + role, + transfer_device, + bucket_size, + metadata_queue, + peer_metadata_queues[role], + result_queue, + ), + ) + for role in ("policy", "rollout") + ] + + try: + for process in processes: + process.start() + metadata = dict(metadata_queue.get(timeout=30) for _ in processes) + ordered_metadata = [metadata["policy"], metadata["rollout"]] + for queue in peer_metadata_queues.values(): + queue.put(ordered_metadata) + for process in processes: + process.join(timeout=60) + + results = dict( + (role, (status, details)) + for role, status, details in ( + result_queue.get(timeout=5) for _ in processes + ) + ) + assert {process.exitcode for process in processes} == {0}, results + assert {status for status, _details in results.values()} == {"success"} + return results["rollout"][1] + finally: + for process in processes: + if process.is_alive(): + process.terminate() + process.join(timeout=5) + if process.is_alive(): + process.kill() + + +class TestCheckpointEngineABC: + def test_cannot_instantiate_abc(self): + with pytest.raises(TypeError): + CheckpointEngine() # type: ignore[abstract] + + def test_subclass_must_implement_all_abstract_methods(self): + class IncompleteEngine(CheckpointEngine): + pass + + with pytest.raises(TypeError): + IncompleteEngine() # type: ignore[abstract] + + +def test_abstract_policy_worker_does_not_enable_checkpoint_engine(): + assert not issubclass(AbstractPolicyWorker, PolicyCheckpointEngineMixin) + assert not hasattr(AbstractPolicyWorker, "checkpoint_engine_rpc") + + +def test_checkpoint_engine_helpers(): + engine = create_checkpoint_engine( + f"{__name__}:_PluginCheckpointEngine", + bucket_size_bytes=16, + engine_kwargs={"marker": "ok"}, + ) + assert isinstance(engine, _PluginCheckpointEngine) + assert (engine.bucket_size, engine.marker) == (16, "ok") + assert not engine.shard_expert_weights + assert engine.get_target_weight_layout() is None + + async def roundtrip(bucket_size): + async def batches(): + for chunk in split_weight_chunks(iter([("weight", tensor)]), bucket_size): + yield [chunk] + + merged = [] + async for batch in merge_weight_chunk_batches(batches()): + merged.extend(batch) + return merged + + tensor = torch.arange(12, dtype=torch.float32).reshape(3, 4) + for bucket_size in (17, 1024): + merged = asyncio.run(roundtrip(bucket_size)) + assert merged[0][0] == "weight" + torch.testing.assert_close(merged[0][1], tensor) + + +def test_create_checkpoint_engine_rejects_unknown_short_name(): + with pytest.raises(ValueError, match="Unknown checkpoint-engine backend 'nixll'"): + create_checkpoint_engine("nixll", bucket_size_bytes=16, engine_kwargs={}) + + +def test_sharded_plugin_requires_target_weight_layout_accessor(): + engine = _PluginCheckpointEngine(bucket_size=16, marker="sharded") + engine.shard_expert_weights = True + + with pytest.raises(NotImplementedError, match="get_target_weight_layout"): + engine.get_target_weight_layout() + + +def test_nixl_checkpoint_engine_rejects_invalid_bucket_size(): + with pytest.raises(ValueError, match="bucket_size must be >= 1"): + NIXLCheckpointEngine(bucket_size=0, device="cpu") + + +@pytest.mark.parametrize( + "rollout_rank,train_world_size,rollout_world_size,error", + [ + (0, 0, 1, "train_world_size must be >= 1"), + (0, 1, 0, "rollout_world_size must be >= 1"), + (-1, 2, 1, "rollout_rank must be in"), + (1, 2, 1, "rollout_rank must be in"), + (0, 1, 2, "train_world_size >= rollout_world_size"), + ], +) +def test_nixl_source_rank_rejects_invalid_topology( + rollout_rank, train_world_size, rollout_world_size, error +): + with pytest.raises(ValueError, match=error): + nixl_mod._source_rank_for_rollout( + rollout_rank, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + ) + + +def test_resolve_nixl_backend_kwargs(): + assert nixl_mod.resolve_nixl_backend_kwargs({}) == ("UCX", None) + assert nixl_mod.resolve_nixl_backend_kwargs( + {"backend_name": "GDS", "backend_init_params": {"device": "nvme0"}} + ) == ("GDS", {"device": "nvme0"}) + + +def test_create_nixl_agent_uses_default_ucx_constructor(monkeypatch): + backend = MagicMock() + nixl_api = SimpleNamespace(nixl_agent=MagicMock(return_value=backend)) + monkeypatch.setattr(nixl_mod.importlib, "import_module", lambda _name: nixl_api) + + assert nixl_mod._create_nixl_agent("agent", "UCX") is backend + nixl_api.nixl_agent.assert_called_once_with("agent") + backend.create_backend.assert_not_called() + + +def test_create_nixl_agent_configures_explicit_backend(monkeypatch): + backend = MagicMock() + config = object() + nixl_api = SimpleNamespace( + nixl_agent=MagicMock(return_value=backend), + nixl_agent_config=MagicMock(return_value=config), + ) + monkeypatch.setattr(nixl_mod.importlib, "import_module", lambda _name: nixl_api) + + assert ( + nixl_mod._create_nixl_agent( + "agent", "UCX", {"engine_config": "MAX_RMA_RAILS=8", "rails": 8} + ) + is backend + ) + nixl_api.nixl_agent.assert_called_once_with("agent", config) + backend.create_backend.assert_called_once_with( + "UCX", {"engine_config": "MAX_RMA_RAILS=8", "rails": "8"} + ) + + +def test_create_nixl_agent_reports_missing_dependency(monkeypatch): + monkeypatch.setattr( + nixl_mod.importlib, + "import_module", + MagicMock(side_effect=ImportError("missing")), + ) + + with pytest.raises(ImportError, match="Install NIXL"): + nixl_mod._create_nixl_agent("agent", "UCX") + + +def test_preinit_nixl_agent_initializes_metadata(monkeypatch): + agent = MagicMock() + create_agent = MagicMock(return_value=agent) + monkeypatch.setattr(nixl_mod, "_create_nixl_agent", create_agent) + monkeypatch.setattr(nixl_mod.uuid, "uuid4", lambda: "id") + + assert ( + nixl_mod.preinit_nixl_agent( + backend_name="GDS", backend_init_params={"path": "/tmp/device"} + ) + is agent + ) + create_agent.assert_called_once_with("preinit-id", "GDS", {"path": "/tmp/device"}) + agent.get_agent_metadata.assert_called_once_with() + + +def test_sync_device_synchronizes_cuda_device(monkeypatch): + synchronize = MagicMock() + monkeypatch.setattr(torch.cuda, "synchronize", synchronize) + + device = torch.device("cuda", 1) + nixl_mod._sync_device(device) + + synchronize.assert_called_once_with(device) + + +@pytest.mark.parametrize("cuda_available", [False, True]) +def test_sync_device_flushes_cpu_staging_copy_when_cuda_is_available( + monkeypatch, cuda_available +): + stream = MagicMock() + monkeypatch.setattr(torch.cuda, "is_available", lambda: cuda_available) + monkeypatch.setattr(torch.cuda, "current_stream", lambda: stream) + + nixl_mod._sync_device(torch.device("cpu")) + + if cuda_available: + stream.synchronize.assert_called_once_with() + else: + stream.synchronize.assert_not_called() + + +def test_maybe_preinit_nixl_checkpoint_engine_defaults_backend(monkeypatch): + calls = [] + fake_nixl = ModuleType("nemo_rl.utils.checkpoint_engines.nixl") + fake_nixl.NIXL_DEFAULT_BACKEND_NAME = "UCX" + + def preinit_nixl_agent(**kwargs): + calls.append(kwargs) + return "agent" + + def resolve_nixl_backend_kwargs(nixl_kwargs): + return ( + nixl_kwargs.get("backend_name", "UCX"), + nixl_kwargs.get("backend_init_params"), + ) + + fake_nixl.preinit_nixl_agent = preinit_nixl_agent + fake_nixl.resolve_nixl_backend_kwargs = resolve_nixl_backend_kwargs + monkeypatch.setitem(sys.modules, "nemo_rl.utils.checkpoint_engines.nixl", fake_nixl) + + assert maybe_preinit_nixl_checkpoint_engine({}) is None + assert ( + maybe_preinit_nixl_checkpoint_engine( + { + "generation": { + "refit_transport": "nixl", + "refit_cfg": {"nixl": {}}, + } + } + ) + == "agent" + ) + assert calls == [{"backend_name": "UCX", "backend_init_params": None}] + + +def test_merge_weight_chunk_batches_uses_aligned_zero_copy_view(): + fp32_w = torch.tensor([7.0, 8.0], dtype=torch.float32) + bucket = torch.zeros(64, dtype=torch.uint8) + offset = 8 + raw = fp32_w.view(torch.uint8) + bucket[offset : offset + fp32_w.nbytes].copy_(raw) + chunk = bucket[offset : offset + fp32_w.nbytes] + meta = TensorMeta( + "fp32_w", + fp32_w.shape, + fp32_w.dtype, + chunk_offset=0, + chunk_size=fp32_w.nbytes, + offset=offset, + ) + + async def run(): + async def batches(): + yield [(meta, chunk)] + + merged = [] + async for batch in merge_weight_chunk_batches(batches()): + merged.extend(batch) + return dict(merged) + + merged = asyncio.run(run()) + torch.testing.assert_close(merged["fp32_w"], fp32_w) + assert ( + merged["fp32_w"].untyped_storage().data_ptr() + == bucket.untyped_storage().data_ptr() + ) + + +def test_merge_weight_chunk_batches_uses_requested_device(monkeypatch): + tensor = torch.arange(8, dtype=torch.float32) + chunks = list(split_weight_chunks(iter([("weight", tensor)]), tensor.nbytes // 2)) + devices = [] + torch_empty = torch.empty + + def record_empty(*args, **kwargs): + devices.append(kwargs["device"]) + return torch_empty(*args, **kwargs) + + monkeypatch.setattr(torch, "empty", record_empty) + + async def run(): + async def batches(): + for chunk in chunks: + yield [chunk] + + return [ + batch + async for batch in merge_weight_chunk_batches(batches(), merge_device="cpu") + ] + + merged = asyncio.run(run()) + + assert devices == ["cpu"] + torch.testing.assert_close(merged[0][0][1], tensor) + + +def test_policy_worker_checkpoint_engine_rpc_runs_weight_send(): + worker = _CheckpointPolicyWorker() + _run_checkpoint_rpc( + worker, + "init_checkpoint_engine", + { + "backend": f"{__name__}:_RecordingCheckpointEngine", + "bucket_size_bytes": 32, + "engine_kwargs": {}, + }, + ) + + assert _run_checkpoint_rpc(worker, "prepare_checkpoint_engine") == { + "bucket_size": 32, + "rank": 3, + } + _run_checkpoint_rpc( + worker, + "init_checkpoint_engine_process_group", + { + "train_world_size": 2, + "rollout_world_size": 1, + "metadata": ["p0", "p1", "g0"], + }, + ) + _run_checkpoint_rpc( + worker, + "send_weights_via_checkpoint_engine", + {"kv_scales": {"scale": 1.0}}, + ) + + assert worker.checkpoint_engine.policy_process_group == { + "worker_rank": 3, + "train_world_size": 2, + "rollout_world_size": 1, + "metadata": ["p0", "p1", "g0"], + } + assert worker.kv_scales == {"scale": 1.0} + assert worker.events == ["prepare", "finalize"] + sent_name, sent_tensor = worker.checkpoint_engine.sent_weights[0] + assert sent_name == "weight" + torch.testing.assert_close(sent_tensor, torch.tensor([1.0, 2.0])) + + _run_checkpoint_rpc(worker, "finalize_checkpoint_engine") + assert worker.checkpoint_engine.finalized + + +def test_policy_worker_checkpoint_engine_rpc_reports_total_memory(monkeypatch): + worker = _CheckpointPolicyWorker() + monkeypatch.setattr(torch.cuda, "current_device", lambda: 2) + get_device_properties = MagicMock(return_value=SimpleNamespace(total_memory=1234)) + monkeypatch.setattr(torch.cuda, "get_device_properties", get_device_properties) + + assert _run_checkpoint_rpc(worker, "checkpoint_engine_total_memory_bytes") == 1234 + get_device_properties.assert_called_once_with(2) + + +def test_policy_worker_checkpoint_engine_rpc_sends_from_running_event_loop(): + worker = _CheckpointPolicyWorker() + _run_checkpoint_rpc( + worker, + "init_checkpoint_engine", + { + "backend": f"{__name__}:_RecordingCheckpointEngine", + "bucket_size_bytes": 32, + "engine_kwargs": {}, + }, + ) + + async def run_send() -> None: + await worker.checkpoint_engine_rpc("send_weights_via_checkpoint_engine") + + asyncio.run(run_send()) + + sent_name, sent_tensor = worker.checkpoint_engine.sent_weights[0] + assert sent_name == "weight" + torch.testing.assert_close(sent_tensor, torch.tensor([1.0, 2.0])) + + +def test_nixl_send_weights_drains_iterator_without_rollout_peer(): + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.next_agent = None + consumed = [] + + def weights(): + consumed.append("started") + yield "weight", torch.tensor([1.0]) + consumed.append("finished") + + asyncio.run(engine.send_weights(weights())) + + assert consumed == ["started", "finished"] + + +def test_nixl_send_weights_aligns_bucket_offsets_for_dtype_views(): + class FakeAgent: + def __init__(self): + self.messages = [] + + def send_message(self, _agent_name, message): + self.messages.append(message) + + async def wait_notification(self, _agent_name, _notify_key): + return None + + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.next_agent = "rollout" + engine.buffers = [ + torch.zeros(64, dtype=torch.uint8), + torch.zeros(64, dtype=torch.uint8), + ] + engine.xfer_descs = ["desc0", "desc1"] + engine.bucket_size = 64 + engine._transfer_device = torch.device("cpu") + engine.agent = FakeAgent() + + weights = iter( + [ + ("bf16_w", torch.tensor([1.0, 2.0, 3.0], dtype=torch.bfloat16)), + ("fp32_w", torch.tensor([7.0], dtype=torch.float32)), + ] + ) + + asyncio.run(engine.send_weights(weights)) + + [message] = engine.agent.messages + assert message["bucket_meta"]["bf16_w"].offset == 0 + assert message["bucket_meta"]["fp32_w"].offset == 8 + assert message["bucket_meta"]["fp32_w"].offset % torch.float32.itemsize == 0 + + +@pytest.mark.parametrize("bucket_size", [17, 32]) +def test_nixl_roundtrip_merges_multibucket_weights_and_alternates_buffers( + monkeypatch, bucket_size +): + sender, receiver = _fake_nixl_pair(monkeypatch, bucket_size) + expected = { + "small": torch.tensor([1.0, 2.0, 3.0], dtype=torch.bfloat16), + "spanning": torch.arange(37, dtype=torch.float32).reshape(37, 1), + "tail": torch.arange(5, dtype=torch.int64), + } + + async def run_roundtrip(): + send_task = asyncio.create_task(sender.send_weights(iter(expected.items()))) + received = {} + async for batch in receiver.receive_weight_batches(): + received.update((name, tensor.clone()) for name, tensor in batch) + await send_task + return received + + received = asyncio.run(run_roundtrip()) + + assert received.keys() == expected.keys() + for name, tensor in expected.items(): + torch.testing.assert_close(received[name], tensor) + + calls = receiver.agent.agent.initialize_calls + assert len(calls) > 2 + assert calls[0][0] == "READ" + assert calls[0][1] is receiver.buffers[1] + assert calls[1][1] is receiver.buffers[0] + assert calls[2][1] is receiver.buffers[1] + assert len(receiver.agent.agent.released_handles) == len(calls) + + +def test_nixl_roundtrip_handles_empty_weight_stream(monkeypatch): + sender, receiver = _fake_nixl_pair(monkeypatch, bucket_size=16) + + async def run_roundtrip(): + send_task = asyncio.create_task(sender.send_weights(iter(()))) + batches = [batch async for batch in receiver.receive_weight_batches()] + await send_task + return batches + + assert asyncio.run(run_roundtrip()) == [] + assert len(receiver.agent.agent.initialize_calls) == 1 + + +@pytest.mark.parametrize( + "transfer_device,bucket_size", + [("cpu", 4096), ("cpu", 8192), ("cuda", 4096), ("cuda", 8192)], +) +def test_nixl_subprocess_roundtrip(transfer_device, bucket_size): + pytest.importorskip("nixl._api") + if not torch.cuda.is_available(): + pytest.skip("CUDA is required to exercise policy-to-transfer buffer copies") + if transfer_device == "cuda": + pytest.importorskip("cupy") + + details = _run_nixl_subprocess_roundtrip(transfer_device, bucket_size) + + assert details["buffer_devices"] == [transfer_device, transfer_device] + assert details["cupy_buffer_count"] == (2 if transfer_device == "cuda" else 0) + assert details["received_devices"] == { + "small": transfer_device, + "spanning": "cpu", + "tail": transfer_device, + } + + +def test_nixl_cuda_transfer_buffer_falls_back_without_cupy(monkeypatch): + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.bucket_size = 16 + engine._transfer_device = torch.device("cuda", 0) + engine._cupy_buffers = [] + engine._cupy_memory_pool = None + engine._uses_torch_cuda_buffers = False + set_device_calls = [] + zeros_calls = [] + + monkeypatch.setattr(torch.cuda, "set_device", set_device_calls.append) + monkeypatch.setattr( + "nemo_rl.utils.checkpoint_engines.nixl.importlib.import_module", + MagicMock(side_effect=ImportError("cupy unavailable")), + ) + + def fake_zeros(*args, **kwargs): + zeros_calls.append((args, kwargs)) + return "buffer" + + monkeypatch.setattr(torch, "zeros", fake_zeros) + + assert engine._allocate_transfer_buffer() == "buffer" + assert set_device_calls == [torch.device("cuda", 0)] + assert zeros_calls == [ + ( + (16,), + {"dtype": torch.uint8, "device": torch.device("cuda", 0)}, + ) + ] + assert engine._cupy_buffers == [] + assert engine._uses_torch_cuda_buffers + + +def test_nixl_cuda_transfer_buffer_uses_private_cupy_pool(monkeypatch): + class Context: + def __enter__(self): + return self + + def __exit__(self, *_args): + return None + + pool = MagicMock() + cupy_buffer = object() + cupy = SimpleNamespace( + uint8="uint8", + zeros=MagicMock(return_value=cupy_buffer), + cuda=SimpleNamespace( + MemoryPool=MagicMock(return_value=pool), + Device=MagicMock(return_value=Context()), + using_allocator=MagicMock(return_value=Context()), + ), + ) + torch_buffer = object() + as_tensor = MagicMock(return_value=torch_buffer) + set_device = MagicMock() + monkeypatch.setattr(nixl_mod.importlib, "import_module", lambda _name: cupy) + monkeypatch.setattr(torch.cuda, "set_device", set_device) + monkeypatch.setattr(torch, "as_tensor", as_tensor) + + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.bucket_size = 16 + engine._transfer_device = torch.device("cuda", 0) + engine._cupy_buffers = [] + engine._cupy_memory_pool = None + engine._uses_torch_cuda_buffers = False + + assert engine._allocate_transfer_buffer() is torch_buffer + set_device.assert_called_once_with(torch.device("cuda", 0)) + cupy.cuda.MemoryPool.assert_called_once_with() + cupy.cuda.Device.assert_called_once_with(0) + cupy.cuda.using_allocator.assert_called_once_with(pool.malloc) + cupy.zeros.assert_called_once_with(16, dtype="uint8") + as_tensor.assert_called_once_with( + cupy_buffer, dtype=torch.uint8, device=torch.device("cuda", 0) + ) + assert engine._cupy_buffers == [cupy_buffer] + assert engine._cupy_memory_pool is pool + + +def test_nixl_release_transfer_buffers_frees_cupy_pool(monkeypatch): + backend = MagicMock() + pool = MagicMock() + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine._transfer_device = torch.device("cpu") + engine.agent = SimpleNamespace(agent=backend) + engine.registration_descs = ["registration-0", "registration-1"] + engine.xfer_descs = ["xfer-0", "xfer-1"] + engine.buffers = [object(), object()] + engine._cupy_buffers = [object(), object()] + engine._cupy_memory_pool = pool + engine._uses_torch_cuda_buffers = False + monkeypatch.setattr(nixl_mod, "_sync_device", MagicMock()) + + engine._release_transfer_buffers() + + assert [call.args for call in backend.deregister_memory.call_args_list] == [ + ("registration-0",), + ("registration-1",), + ] + assert engine.registration_descs == [] + assert engine.xfer_descs == [] + assert engine.buffers == [] + assert engine._cupy_buffers == [] + pool.free_all_blocks.assert_called_once_with() + assert engine._cupy_memory_pool is None + + +def test_nixl_release_torch_cuda_buffers_empties_cache(monkeypatch): + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine._transfer_device = torch.device("cuda", 0) + engine.agent = SimpleNamespace(agent=MagicMock()) + engine.registration_descs = [] + engine.xfer_descs = [] + engine.buffers = [] + engine._cupy_buffers = [] + engine._cupy_memory_pool = None + engine._uses_torch_cuda_buffers = True + empty_cache = MagicMock() + monkeypatch.setattr(nixl_mod, "_sync_device", MagicMock()) + monkeypatch.setattr(torch.cuda, "empty_cache", empty_cache) + + engine._release_transfer_buffers() + + empty_cache.assert_called_once_with() + assert not engine._uses_torch_cuda_buffers + + +def test_nixl_finalize_disconnects_peers(): + class FakeAgent: + def __init__(self): + self.removed = [] + + def remove_remote_agent(self, agent_name): + self.removed.append(agent_name) + + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.agent = FakeAgent() + engine.prev_agent = "policy" + engine.next_agent = "rollout" + engine._target_weight_layout = {"layer": {}} + engine.release_after_refit = False + + engine.finalize() + + assert engine.agent.removed == ["policy", "rollout"] + assert engine.prev_agent is None + assert engine.next_agent is None + assert engine.get_target_weight_layout() is None + + +def test_nixl_release_after_refit_deregisters_and_reregisters_buffers(monkeypatch): + class FakeBackend: + def __init__(self): + self.registered = [] + self.deregistered = [] + + def register_memory(self, buffer): + registration = f"registration-{len(self.registered)}" + self.registered.append((registration, buffer)) + return registration + + def get_xfer_descs(self, buffer): + return f"xfer-{len(self.registered)}-{buffer.numel()}" + + def deregister_memory(self, registration): + self.deregistered.append(registration) + + class FakeAgent: + instances = [] + + def __init__(self, backend_name, backend_init_params): + self.backend_name = backend_name + self.backend_init_params = backend_init_params + self.agent = FakeBackend() + self.instances.append(self) + + def get_agent_metadata(self): + return {"agent": len(self.instances)} + + monkeypatch.setattr(nixl_mod, "NixlAgent", FakeAgent) + engine = NIXLCheckpointEngine( + bucket_size=16, + device="cpu", + backend_name="UCX", + backend_init_params={"device_list": "mlx5_0:1"}, + release_after_refit=True, + ) + + first_agent = engine.agent + assert engine.prepare() == {"agent": 1} + assert len(first_agent.agent.registered) == 2 + assert len(engine.buffers) == len(engine.registration_descs) == 2 + + engine.finalize() + + assert first_agent.agent.deregistered == ["registration-0", "registration-1"] + assert engine.agent is first_agent + assert engine.buffers == [] + assert engine.registration_descs == [] + assert engine.xfer_descs == [] + + assert engine.prepare() == {"agent": 1} + assert engine.agent is first_agent + assert len(engine.agent.agent.registered) == 4 + + +def test_nixl_default_finalize_retains_registered_resources(monkeypatch): + agent = MagicMock() + agent.get_agent_metadata.return_value = {"agent": "retained"} + agent.agent.register_memory.side_effect = ["registration-0", "registration-1"] + agent.agent.get_xfer_descs.side_effect = ["xfer-0", "xfer-1"] + monkeypatch.setattr(nixl_mod, "NixlAgent", lambda *_args: agent) + engine = NIXLCheckpointEngine(bucket_size=16, device="cpu") + + engine.prepare() + buffers = engine.buffers + engine.finalize() + + assert engine.agent is agent + assert engine.buffers is buffers + agent.agent.deregister_memory.assert_not_called() + + +def test_nixl_agent_binds_zmq_socket_atomically(monkeypatch): + class FakeNixlBackend: + def get_agent_metadata(self): + return {"backend": "metadata"} + + class FakePushContext: + pass + + class FakePullSocket: + def __init__(self): + self.bind_endpoints = [] + + def bind_to_random_port(self, endpoint): + self.bind_endpoints.append(endpoint) + return 45678 + + class FakePullContext: + def __init__(self, socket): + self._socket = socket + + def socket(self, socket_type): + assert socket_type == nixl_mod.zmq.PULL + return self._socket + + pull_socket = FakePullSocket() + monkeypatch.setattr( + nixl_mod, + "_create_nixl_agent", + lambda agent_name, backend_name, backend_init_params: FakeNixlBackend(), + ) + monkeypatch.setattr(nixl_mod.ray.util, "get_node_ip_address", lambda: "10.10.0.12") + monkeypatch.setattr(nixl_mod.zmq, "Context", lambda: FakePushContext()) + monkeypatch.setattr( + nixl_mod.zmq.asyncio, "Context", lambda: FakePullContext(pull_socket) + ) + + agent = nixl_mod.NixlAgent() + + assert agent.listen_port == 45678 + assert pull_socket.bind_endpoints == ["tcp://10.10.0.12"] + assert agent.get_agent_metadata()["zmq_port"] == 45678 + + +def test_nixl_agent_connects_sends_and_removes_remote(): + backend = MagicMock() + backend.add_remote_agent.return_value = b"remote" + socket = MagicMock() + context = MagicMock() + context.socket.return_value = socket + agent = nixl_mod.NixlAgent.__new__(nixl_mod.NixlAgent) + agent.agent_name = "local" + agent.agent = backend + agent.zmq_client_context = context + agent.zmq_clients = {} + + metadata = { + "agent_metadata": b"metadata", + "zmq_ip": "10.0.0.2", + "zmq_port": 1234, + } + assert agent.add_remote_agent(metadata) == "remote" + socket.connect.assert_called_once_with("tcp://10.0.0.2:1234") + + message = {"key": "value"} + agent.send_message("remote", message) + socket.send_pyobj.assert_called_once_with(("local", message), nixl_mod.zmq.DONTWAIT) + + agent.remove_remote_agent("remote") + backend.remove_remote_agent.assert_called_once_with("remote") + socket.close.assert_called_once_with(linger=0) + assert agent.zmq_clients == {} + + +def test_nixl_agent_reads_messages_and_progresses_backend(): + backend = MagicMock() + socket = SimpleNamespace( + recv_pyobj=AsyncMock( + side_effect=[nixl_mod.zmq.Again(), ("remote", {"value": 1})] + ) + ) + agent = nixl_mod.NixlAgent.__new__(nixl_mod.NixlAgent) + agent.agent = backend + agent.socket = socket + agent.messages = defaultdict(deque) + + assert asyncio.run(agent.read_message("remote")) == {"value": 1} + assert backend.progress.call_count == 2 + assert socket.recv_pyobj.await_count == 2 + + +def test_nixl_agent_waits_for_matching_notification(): + backend = MagicMock() + backend.get_new_notifs.side_effect = [ + {b"remote": [b"other", b"wanted"]}, + ] + agent = nixl_mod.NixlAgent.__new__(nixl_mod.NixlAgent) + agent.agent = backend + agent.notifications = defaultdict(deque) + + asyncio.run(agent.wait_notification("remote", b"wanted")) + + assert list(agent.notifications["remote"]) == [b"other"] + backend.progress.assert_called_once_with() + backend.get_new_notifs.assert_called_once_with() + + +def test_nixl_wait_read_completes_and_releases_handle(): + backend = MagicMock() + backend.check_xfer_state.side_effect = ["IN_PROGRESS", "DONE"] + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.agent = SimpleNamespace(agent=backend) + + asyncio.run(engine._wait_read("handle", "policy")) + + assert backend.progress.call_count == 2 + assert backend.check_xfer_state.call_count == 2 + backend.release_xfer_handle.assert_called_once_with("handle") + + +def test_nixl_wait_read_reports_transfer_error(): + backend = MagicMock() + backend.check_xfer_state.return_value = "ERR" + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.agent = SimpleNamespace(agent=backend) + + with pytest.raises(RuntimeError, match="read from policy failed"): + asyncio.run(engine._wait_read("handle", "policy")) + backend.release_xfer_handle.assert_not_called() + + +def test_nixl_receive_requires_initialized_rollout_process_group(): + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.prev_agent = None + + async def receive_one(): + return await anext(engine._receive_weight_chunk_batches()) + + with pytest.raises(RuntimeError, match="rollout process group is not initialized"): + asyncio.run(receive_one()) + + +def test_nixl_receive_reports_transfer_start_error(): + backend = MagicMock() + backend.initialize_xfer.return_value = "handle" + backend.transfer.return_value = "ERR" + agent = SimpleNamespace( + agent=backend, + read_message=AsyncMock( + return_value={ + "remote_descs": "remote", + "notify_key": b"key", + "bucket_meta": {}, + "is_last": True, + } + ), + ) + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.prev_agent = "policy" + engine.agent = agent + engine.buffers = [torch.zeros(8, dtype=torch.uint8) for _ in range(2)] + engine.xfer_descs = ["local-0", "local-1"] + + async def receive_one(): + return await anext(engine._receive_weight_chunk_batches()) + + with pytest.raises(RuntimeError, match="failed to start"): + asyncio.run(receive_one()) + backend.initialize_xfer.assert_called_once_with( + "READ", "local-1", "remote", "policy", b"key" + ) + + +def test_nixl_process_group_uses_parallel_policy_to_rollout_topology(): + def engine_with_agent(): + engine = NIXLCheckpointEngine.__new__(NIXLCheckpointEngine) + engine.prev_agent = None + engine.next_agent = None + engine._target_weight_layout = None + engine.shard_expert_weights = True + engine.agent = MagicMock() + engine.agent.add_remote_agent.side_effect = lambda metadata: metadata["name"] + return engine + + rollout_0_layout = {"layer.0": {}} + rollout_1_layout = {"layer.1": {}} + rollout_2_layout = {"layer.2": {}} + metadata = [ + {"name": "policy-0"}, + {"name": "policy-1"}, + {"name": "policy-2"}, + {"name": "policy-3"}, + {"name": "rollout-0", "weight_layout": rollout_0_layout}, + {"name": "rollout-1", "weight_layout": rollout_1_layout}, + {"name": "rollout-2", "weight_layout": rollout_2_layout}, + ] + + policy_rank_0 = engine_with_agent() + policy_rank_0.init_policy_process_group( + worker_rank=0, + train_world_size=4, + rollout_world_size=3, + metadata=metadata, + ) + assert policy_rank_0.next_agent == "rollout-0" + assert policy_rank_0.get_target_weight_layout() is rollout_0_layout + + policy_rank_1 = engine_with_agent() + policy_rank_1.init_policy_process_group( + worker_rank=1, + train_world_size=4, + rollout_world_size=3, + metadata=metadata, + ) + assert policy_rank_1.next_agent == "rollout-1" + assert policy_rank_1.get_target_weight_layout() is rollout_1_layout + + policy_rank_3 = engine_with_agent() + policy_rank_3.init_policy_process_group( + worker_rank=3, + train_world_size=4, + rollout_world_size=3, + metadata=metadata, + ) + assert policy_rank_3.next_agent is None + assert policy_rank_3.get_target_weight_layout() is None + policy_rank_3.agent.add_remote_agent.assert_not_called() + + rollout_rank_0 = engine_with_agent() + rollout_rank_0.init_rollout_process_group( + rollout_rank=0, + train_world_size=4, + rollout_world_size=3, + metadata=metadata, + ) + assert (rollout_rank_0.prev_agent, rollout_rank_0.next_agent) == ("policy-0", None) + + rollout_rank_1 = engine_with_agent() + rollout_rank_1.init_rollout_process_group( + rollout_rank=1, + train_world_size=4, + rollout_world_size=3, + metadata=metadata, + ) + assert (rollout_rank_1.prev_agent, rollout_rank_1.next_agent) == ("policy-1", None) + + rollout_rank_2 = engine_with_agent() + rollout_rank_2.init_rollout_process_group( + rollout_rank=2, + train_world_size=4, + rollout_world_size=3, + metadata=metadata, + ) + assert (rollout_rank_2.prev_agent, rollout_rank_2.next_agent) == ( + "policy-2", + None, + ) diff --git a/tests/unit/utils/test_weight_transfer_stream.py b/tests/unit/utils/test_weight_transfer_stream.py index 4ab661fc657..0a197da5cee 100644 --- a/tests/unit/utils/test_weight_transfer_stream.py +++ b/tests/unit/utils/test_weight_transfer_stream.py @@ -24,7 +24,7 @@ import torch import zstandard -from nemo_rl.models.generation.vllm.config import VllmRefitConfig +from nemo_rl.models.generation.vllm.config import VllmSparseRefitConfig from nemo_rl.utils import ( weight_transfer_http, weight_transfer_stream, @@ -62,8 +62,8 @@ def _refit_config( s3_transfer_workers: int = 32, zmq_transfer_workers: int = 4, zmq_retries: int = 3, -) -> VllmRefitConfig: - return VllmRefitConfig( +) -> VllmSparseRefitConfig: + return VllmSparseRefitConfig( delta_compression={ "encoding": encoding, "sparse_bucket_size_bytes": bucket_bytes, @@ -567,7 +567,7 @@ def test_s3_manifest_transport_validates_configuration(monkeypatch) -> None: ) tracker.refit_config = _refit_config() - with pytest.raises(RuntimeError, match="refit_cfg.storage.s3_bucket"): + with pytest.raises(RuntimeError, match="refit_cfg.sparse.storage.s3_bucket"): weight_transfer_stream.stream_sparse_delta_payloads_via_s3_manifest( refit_targets=["http://receiver"], **kwargs ) diff --git a/tests/unit/weight_sync/test_checkpoint_engine_config.py b/tests/unit/weight_sync/test_checkpoint_engine_config.py new file mode 100644 index 00000000000..f30c7d9852f --- /dev/null +++ b/tests/unit/weight_sync/test_checkpoint_engine_config.py @@ -0,0 +1,81 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for checkpoint-engine configuration selection.""" + +import pytest + +from nemo_rl.weight_sync.checkpoint_engine_config import ( + checkpoint_engine_refit_config, +) + + +@pytest.mark.parametrize( + "generation_config", + [{}, {"refit_transport": None}, {"refit_transport": "vllm_zmq_sparse"}], +) +def test_checkpoint_engine_refit_config_returns_none_for_other_transports( + generation_config, +): + assert checkpoint_engine_refit_config(generation_config) is None + + +def test_checkpoint_engine_refit_config_resolves_nixl_defaults(): + generation_config = {"refit_transport": "nixl", "refit_cfg": None} + + assert checkpoint_engine_refit_config(generation_config) == { + "backend": "nixl", + "update_weights_bucket_memory_ratio": 0.05, + "engine_kwargs": { + "nixl": { + "device": "cuda", + "backend_name": "UCX", + "backend_init_params": None, + "release_after_refit": False, + "shard_expert_weights": False, + } + }, + } + + +def test_checkpoint_engine_refit_config_resolves_custom_backend_scope(): + backend = "package.engine:CustomCheckpointEngine" + generation_config = { + "refit_transport": backend, + "refit_cfg": { + backend: { + "update_weights_bucket_memory_ratio": 0.1, + "release_after_refit": True, + "custom_option": 7, + } + }, + } + + assert checkpoint_engine_refit_config(generation_config) == { + "backend": backend, + "update_weights_bucket_memory_ratio": 0.1, + "engine_kwargs": {backend: {"release_after_refit": True, "custom_option": 7}}, + } + + +def test_checkpoint_engine_refit_config_rejects_legacy_block(): + with pytest.raises(ValueError, match="checkpoint_engine was replaced"): + checkpoint_engine_refit_config( + {"checkpoint_engine": {"enabled": True, "backend": "nixl"}} + ) + + +def test_checkpoint_engine_refit_config_rejects_unknown_selector(): + with pytest.raises(ValueError, match="Unknown vLLM refit transport"): + checkpoint_engine_refit_config({"refit_transport": "nixll"}) diff --git a/tests/unit/weight_sync/test_checkpoint_engine_weight_synchronizer.py b/tests/unit/weight_sync/test_checkpoint_engine_weight_synchronizer.py new file mode 100644 index 00000000000..8a02257e8fa --- /dev/null +++ b/tests/unit/weight_sync/test_checkpoint_engine_weight_synchronizer.py @@ -0,0 +1,364 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for checkpoint-engine weight synchronization and factory routing.""" + +from unittest.mock import MagicMock, patch + +import pytest + +from nemo_rl.models.generation.constants import ( + MEGATRON_BACKEND, + SGLANG_BACKEND, + VLLM_BACKEND, +) +from nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer import ( + CheckpointEngineWeightSynchronizer, + _ordered_generation_metadata, + _sort_ranked_metadata, +) +from nemo_rl.weight_sync.factory import create_weight_synchronizer + + +def _mock_policy(**overrides): + policy = MagicMock() + policy.offload_before_refit.return_value = None + policy.offload_after_refit.return_value = None + policy.prepare_refit_info.return_value = {"layer_0": {"shape": [4096, 4096]}} + policy.stream_weights_via_ipc_zmq.return_value = [MagicMock()] + policy.stream_weights_via_http.return_value = [MagicMock()] + policy.broadcast_weights_for_collective.return_value = [MagicMock()] + policy.init_collective.return_value = [MagicMock()] + policy.get_free_memory_bytes.return_value = 1024**3 # 1 GB + for k, v in overrides.items(): + setattr(policy, k, v) + return policy + + +def _mock_generation(**overrides): + gen = MagicMock() + gen.cfg = {} + gen.prepare_for_generation.return_value = True + gen.finish_generation.return_value = True + gen.prepare_refit_info.return_value = None + gen.update_weights_via_ipc_zmq.return_value = [MagicMock()] + gen.update_weights_from_collective.return_value = [MagicMock()] + gen.get_rollout_engine_urls.return_value = ["http://localhost:30000"] + gen.init_collective.return_value = [MagicMock()] + for k, v in overrides.items(): + setattr(gen, k, v) + return gen + + +def _checkpoint_engine_cfg( + *, + release_after_refit=False, + backend="test_backend", + bucket_memory_ratio=0.05, + device="cpu", +): + return { + "backend": backend, + "update_weights_bucket_memory_ratio": bucket_memory_ratio, + "engine_kwargs": { + backend: { + "device": device, + "release_after_refit": release_after_refit, + } + }, + } + + +def _nixl_refit_cfg(*, release_after_refit=False): + return { + "refit_transport": "nixl", + "refit_cfg": { + "nixl": { + "device": "cpu", + "release_after_refit": release_after_refit, + } + }, + "vllm_cfg": {"async_engine": False}, + } + + +class _CheckpointWorkerGroup: + def __init__(self): + self.workers = [object(), object(), object(), object()] + self.calls = [] + + def run_all_workers_single_data(self, method_name, **kwargs): + self.calls.append((method_name, kwargs["checkpoint_method"])) + return [kwargs["checkpoint_method"]] + + def run_all_workers_multiple_data(self, method_name, **kwargs): + self.calls.append( + ( + method_name, + kwargs["common_kwargs"]["checkpoint_method"], + kwargs["method_args"], + ) + ) + return ["generation-init"] + + +def _checkpoint_sync( + mock_ray, + *, + async_engine=False, + update_success=True, + release_after_refit=False, + cycles=1, + checkpoint_engine_config=None, +): + # One return value per ray.get() call, in order: + # 1. total GPU memory (policy + generation) + # 2. init_checkpoint_engine (policy + generation) + # 3. prepare_checkpoint_engine (policy refs first, then generation refs; + # _CheckpointWorkerGroup returns a single ref per side here) + # 4. init_checkpoint_engine_process_group (policy + generation) + # 5. send + update_weights_from_checkpoint_engine + # 6. finalize_checkpoint_engine (policy + generation) + mock_ray.get.side_effect = [[80 * 1024**3, [80 * 1024**3]]] + [ + item + for _ in range(cycles) + for item in ( + [], + [["policy-0", "policy-1"], "generation-0", ["generation-1"]], + [], + ["policy-send", update_success], + [], + ) + ] + policy = _mock_policy() + policy.worker_group = _CheckpointWorkerGroup() + checkpoint_engine_config = checkpoint_engine_config or _checkpoint_engine_cfg( + release_after_refit=release_after_refit + ) + gen = _mock_generation(cfg={"vllm_cfg": {"async_engine": async_engine}}) + gen.dp_size = 2 + gen.worker_group = _CheckpointWorkerGroup() + return CheckpointEngineWeightSynchronizer(policy, gen, checkpoint_engine_config) + + +class TestCheckpointEngineWeightSynchronizer: + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_bucket_uses_minimum_total_memory_and_is_cached(self, mock_ray, capsys): + config = _checkpoint_engine_cfg(bucket_memory_ratio=0.125) + sync = _checkpoint_sync(mock_ray, checkpoint_engine_config=config) + mock_ray.get.side_effect = None + mock_ray.get.return_value = [96 * 1024**3, [64 * 1024**3, 80 * 1024**3]] + + assert sync._resolve_bucket_size_bytes() == 8192 * 1024**2 + assert sync._resolve_bucket_size_bytes() == 8192 * 1024**2 + mock_ray.get.assert_called_once() + assert sync._policy.worker_group.calls == [ + ("checkpoint_engine_rpc", "checkpoint_engine_total_memory_bytes") + ] + assert sync._generation.worker_group.calls == [ + ("checkpoint_engine_rpc", "checkpoint_engine_total_memory_bytes") + ] + assert "8192 MiB per buffer" in capsys.readouterr().out + + @pytest.mark.parametrize("memory_ratio", ["invalid", 0, 1]) + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_bucket_rejects_invalid_ratio(self, mock_ray, memory_ratio): + config = _checkpoint_engine_cfg(bucket_memory_ratio=memory_ratio) + sync = _checkpoint_sync(mock_ray, checkpoint_engine_config=config) + + with pytest.raises(ValueError, match="update_weights_bucket_memory_ratio"): + sync._resolve_bucket_size_bytes() + mock_ray.get.assert_not_called() + + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_bucket_rejects_sub_mibibyte_result(self, mock_ray): + config = _checkpoint_engine_cfg(bucket_memory_ratio=0.05) + sync = _checkpoint_sync(mock_ray, checkpoint_engine_config=config) + mock_ray.get.side_effect = None + mock_ray.get.return_value = [8 * 1024**2, [8 * 1024**2]] + + with pytest.raises(ValueError, match="less than 1 MiB"): + sync._resolve_bucket_size_bytes() + + def test_sort_ranked_metadata_orders_by_rank(self): + metadata = [{"rank": 2}, {"rank": 0}, {"rank": 1}] + + assert _sort_ranked_metadata(metadata) == [ + {"rank": 0}, + {"rank": 1}, + {"rank": 2}, + ] + + def test_ordered_generation_metadata_handles_dp_groups_with_colliding_ranks(self): + # Two vLLM DP groups (engines), each reporting engine-local ranks 0/1 that + # collide across groups; collective_rpc may return them out of local order. + # The result must be global rollout-rank order: [g0r0, g0r1, g1r0, g1r1]. + generation_results = [ + [{"rank": 1, "id": "g0r1"}, {"rank": 0, "id": "g0r0"}], + [{"rank": 1, "id": "g1r1"}, {"rank": 0, "id": "g1r0"}], + ] + + ordered = _ordered_generation_metadata(generation_results) + + assert [m["id"] for m in ordered] == ["g0r0", "g0r1", "g1r0", "g1r1"] + # A single global sort over colliding ranks would instead interleave the + # groups ([g0r0, g1r0, g0r1, g1r1]) and mis-pair policy<->rollout workers. + + def test_ordered_generation_metadata_single_group(self): + generation_results = [[{"rank": 1, "id": "r1"}, {"rank": 0, "id": "r0"}]] + + ordered = _ordered_generation_metadata(generation_results) + + assert [m["id"] for m in ordered] == ["r0", "r1"] + + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_sync_weights_runs_checkpoint_engine_lifecycle(self, mock_ray): + sync = _checkpoint_sync(mock_ray) + + sync.init_communicator() + sync.sync_weights(kv_scales={"kv": 1.0}) + + assert not sync.is_stale + sync._policy.prepare_refit_info.assert_called_once() + sync._generation.prepare_refit_info.assert_called_once() + assert ( + "checkpoint_engine_rpc", + "send_weights_via_checkpoint_engine", + ) in sync._policy.worker_group.calls + assert ( + "checkpoint_engine_rpc", + "update_weights_from_checkpoint_engine", + ) in sync._generation.worker_group.calls + assert sync._generation.worker_group.calls[3][2] == [ + (0, 2, 2, ["policy-0", "policy-1", "generation-0", "generation-1"]), + (2, 2, 2, ["policy-0", "policy-1", "generation-0", "generation-1"]), + ] + sync.mark_stale() + assert sync.is_stale + sync.shutdown() + assert sync._generation.worker_group.calls[-1] == ( + "checkpoint_engine_rpc", + "finalize_checkpoint_engine", + ) + assert sync._policy.worker_group.calls[-1] == ( + "checkpoint_engine_rpc", + "finalize_checkpoint_engine", + ) + + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_release_after_refit_reprepares_each_sync(self, mock_ray): + sync = _checkpoint_sync(mock_ray, release_after_refit=True, cycles=2) + + sync.init_communicator() + sync.sync_weights() + assert not sync._checkpoint_engine_ready + + sync.sync_weights() + assert not sync._checkpoint_engine_ready + assert ( + sync._policy.worker_group.calls.count( + ("checkpoint_engine_rpc", "prepare_checkpoint_engine") + ) + == 2 + ) + assert ( + sync._policy.worker_group.calls.count( + ("checkpoint_engine_rpc", "finalize_checkpoint_engine") + ) + == 2 + ) + + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_sync_weights_does_not_run_colocated_phase_transitions(self, mock_ray): + sync = _checkpoint_sync(mock_ray) + + sync.init_communicator() + sync.sync_weights() + + sync._policy.offload_before_refit.assert_not_called() + sync._policy.offload_after_refit.assert_not_called() + sync._policy.prepare_for_training.assert_not_called() + sync._generation.prepare_for_generation.assert_not_called() + + @patch("nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer.ray") + def test_sync_weights_raises_when_generation_update_fails(self, mock_ray): + sync = _checkpoint_sync(mock_ray, async_engine=True, update_success=False) + + sync.init_communicator() + with pytest.raises(RuntimeError, match="Weight transfer failed"): + sync.sync_weights() + + assert sync.is_stale + assert sync._generation.worker_group.calls[-1] == ( + "checkpoint_engine_rpc_async", + "update_weights_from_checkpoint_engine", + ) + sync.shutdown() + assert sync._generation.worker_group.calls[-1] == ( + "checkpoint_engine_rpc_async", + "finalize_checkpoint_engine", + ) + assert ( + sync._generation.worker_group.calls[0][0] == "checkpoint_engine_rpc_async" + ) + assert sync._policy.worker_group.calls[-1] == ( + "checkpoint_engine_rpc", + "finalize_checkpoint_engine", + ) + + +class TestCheckpointEngineFactory: + @pytest.mark.parametrize( + ("backend", "colocated", "expected"), + [ + (VLLM_BACKEND, False, CheckpointEngineWeightSynchronizer), + (VLLM_BACKEND, True, ValueError), + (SGLANG_BACKEND, False, NotImplementedError), + (MEGATRON_BACKEND, False, NotImplementedError), + ], + ) + def test_checkpoint_engine_factory_routing(self, backend, colocated, expected): + policy = _mock_policy(cfg={}) + gen = _mock_generation(cfg=_nixl_refit_cfg()) + if isinstance(expected, type) and issubclass(expected, Exception): + with pytest.raises(expected): + create_weight_synchronizer( + policy=policy, + generation=gen, + generation_backend=backend, + colocated=colocated, + ) + return + assert isinstance( + create_weight_synchronizer( + policy=policy, + generation=gen, + generation_backend=backend, + colocated=colocated, + ), + expected, + ) + + @pytest.mark.parametrize("cfg", [{"megatron_cfg": {"enabled": False}}, {}]) + def test_checkpoint_engine_accepts_non_megatron_policy(self, cfg): + gen = _mock_generation(cfg=_nixl_refit_cfg()) + assert isinstance( + create_weight_synchronizer( + policy=_mock_policy(cfg=cfg), + generation=gen, + generation_backend=VLLM_BACKEND, + colocated=False, + ), + CheckpointEngineWeightSynchronizer, + ) diff --git a/tests/unit/weight_sync/test_vllm_remote_sparse_weight_synchronizer.py b/tests/unit/weight_sync/test_vllm_remote_sparse_weight_synchronizer.py index 0c6f5a6c461..d55acb5a055 100644 --- a/tests/unit/weight_sync/test_vllm_remote_sparse_weight_synchronizer.py +++ b/tests/unit/weight_sync/test_vllm_remote_sparse_weight_synchronizer.py @@ -71,8 +71,10 @@ def _valid_config() -> dict: return { "refit_transport": "vllm_s3_sparse", "refit_cfg": { - "delta_compression": {"encoding": "overwrite"}, - "storage": {"s3_bucket": "bucket"}, + "sparse": { + "delta_compression": {"encoding": "overwrite"}, + "storage": {"s3_bucket": "bucket"}, + } }, "vllm_cfg": {"precision": "bfloat16", "kv_cache_dtype": "auto"}, } @@ -87,8 +89,10 @@ def test_validate_remote_sparse_refit_accepts_supported_scope(): == "s3" ) assert config["refit_cfg"] == VllmRefitConfig( - delta_compression={"encoding": "overwrite"}, - storage={"s3_bucket": "bucket"}, + sparse={ + "delta_compression": {"encoding": "overwrite"}, + "storage": {"s3_bucket": "bucket"}, + } ) @@ -98,7 +102,7 @@ def test_validate_remote_sparse_refit_accepts_supported_scope(): ({"refit_transport": "unknown"}, {}), ({}, {"colocated": True}), ({}, {"megatron_enabled": False}), - ({"refit_cfg": {"storage": {"s3_bucket": None}}}, {}), + ({"refit_cfg": {"sparse": {"storage": {"s3_bucket": None}}}}, {}), ({"quant_cfg": "fp8"}, {}), ({"vllm_cfg": {"precision": "fp8", "kv_cache_dtype": "auto"}}, {}), ( diff --git a/tests/unit/weight_sync/test_weight_synchronizer.py b/tests/unit/weight_sync/test_weight_synchronizer.py index edec20b6c0a..0db81a29f7f 100644 --- a/tests/unit/weight_sync/test_weight_synchronizer.py +++ b/tests/unit/weight_sync/test_weight_synchronizer.py @@ -57,6 +57,7 @@ def _mock_policy(**overrides): def _mock_generation(**overrides): gen = MagicMock() + gen.cfg = {} gen.prepare_for_generation.return_value = True gen.finish_generation.return_value = True gen.prepare_refit_info.return_value = None diff --git a/tools/nixl_elastic_rollout_demo.py b/tools/nixl_elastic_rollout_demo.py new file mode 100644 index 00000000000..1fb75e5d913 --- /dev/null +++ b/tools/nixl_elastic_rollout_demo.py @@ -0,0 +1,566 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Demonstrate NIXL checkpoint-engine refit after rollout actors change. + +The current NIXL checkpoint engine pairs policy rank ``i`` with rollout rank +``i``. This demo creates one policy endpoint for every possible rollout rank, +then adds and removes rollout endpoints between phases. Each phase exchanges +fresh NIXL metadata, rebuilds the process groups, performs a full synthetic +weight refit, and verifies the active rollout endpoints before continuing. +""" + +from __future__ import annotations + +import argparse +import asyncio +import contextlib +import json +import os +import time +from collections.abc import Iterable +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + import torch + + +@dataclass(frozen=True) +class EndpointHandle: + rank: int + generation: int + node_id: str + node_ip: str + actor: Any + + +def _env_subset() -> dict[str, str]: + prefixes = ("UCX_", "NIXL_", "MELLANOX_", "NVIDIA_") + names = { + "CUDA_VISIBLE_DEVICES", + "LD_LIBRARY_PATH", + "PYTHONPATH", + "RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES", + } + return { + key: value + for key, value in os.environ.items() + if key in names or key.startswith(prefixes) + } + + +def _node_label(node: dict[str, Any]) -> str: + return f"{node['NodeManagerAddress']}:{node['NodeID'][:8]}" + + +def _alive_nodes() -> list[dict[str, Any]]: # pragma: no cover + import ray + + nodes = [node for node in ray.nodes() if node.get("Alive")] + return sorted(nodes, key=lambda node: node["NodeManagerAddress"]) + + +def _wait_for_nodes( + min_nodes: int, timeout_s: int +) -> list[dict[str, Any]]: # pragma: no cover + deadline = time.time() + timeout_s + while time.time() < deadline: + nodes = _alive_nodes() + if len(nodes) >= min_nodes: + return nodes + time.sleep(2) + nodes = _alive_nodes() + raise RuntimeError( + f"Timed out waiting for {min_nodes} live Ray nodes; saw " + f"{len(nodes)}: {[_node_label(node) for node in nodes]}" + ) + + +def _actor_options(node: dict[str, Any], *, use_gpu: bool) -> dict[str, Any]: + from ray.util.scheduling_strategies import NodeAffinitySchedulingStrategy + + return { + "num_cpus": 1, + "num_gpus": 1 if use_gpu else 0, + "runtime_env": {"env_vars": _env_subset()}, + "scheduling_strategy": NodeAffinitySchedulingStrategy( + node_id=node["NodeID"], + soft=False, + ), + } + + +def _parse_sequence(value: str) -> list[int]: + sequence = [int(item.strip()) for item in value.split(",") if item.strip()] + if not sequence: + raise ValueError("rollout sequence must contain at least one count") + if min(sequence) < 1: + raise ValueError("rollout counts must be positive") + return sequence + + +def _phase_delta( + active_ranks: Iterable[int], target_count: int +) -> tuple[list[int], list[int]]: + current = set(active_ranks) + target = set(range(target_count)) + return sorted(target - current), sorted(current - target) + + +def _build_engine_kwargs(args: argparse.Namespace) -> dict[str, Any]: + return { + "device": args.device, + "backend_name": args.nixl_backend_name, + "backend_init_params": { + "ucx_error_handling_mode": args.ucx_error_handling_mode, + }, + "release_after_refit": True, + } + + +def _tensor_summary( + weights: Iterable[tuple[str, "torch.Tensor"]], +) -> dict[str, dict[str, Any]]: + import torch + + return { + name: { + "shape": list(tensor.shape), + "dtype": str(tensor.dtype), + "sum": int(tensor.detach().cpu().sum(dtype=torch.int64).item()), + } + for name, tensor in weights + } + + +class ElasticNixlEndpoint: # pragma: no cover + def __init__( + self, + *, + label: str, + rank: int, + backend: str, + bucket_size_bytes: int, + engine_kwargs: dict[str, Any], + device: str, + ) -> None: + import ray + import torch + + from nemo_rl.utils.checkpoint_engines.base import create_checkpoint_engine + + self.label = label + self.rank = rank + self.hostname = os.uname().nodename + self.node_ip = ray.util.get_node_ip_address().strip("[]") + self.device = torch.device(device) + if self.device.type == "cuda": + torch.cuda.set_device(0) + torch.empty(1, device="cuda").fill_(1) + torch.cuda.synchronize() + + self.engine = create_checkpoint_engine( + backend, + bucket_size_bytes=bucket_size_bytes, + engine_kwargs=engine_kwargs, + ) + + def report(self) -> dict[str, Any]: + return { + "label": self.label, + "rank": self.rank, + "hostname": self.hostname, + "node_ip": self.node_ip, + "pid": os.getpid(), + "device": str(self.device), + "cuda_visible_devices": os.environ.get("CUDA_VISIBLE_DEVICES"), + "ucx_net_devices": os.environ.get("UCX_NET_DEVICES"), + "ucx_tls": os.environ.get("UCX_TLS"), + "ucx_max_rndv_rails": os.environ.get("UCX_MAX_RNDV_RAILS"), + } + + def prepare(self) -> Any: + return self.engine.prepare() + + def init_policy( + self, + *, + metadata: list[Any], + train_world_size: int, + rollout_world_size: int, + ) -> None: + self.engine.init_policy_process_group( + worker_rank=self.rank, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + metadata=metadata, + ) + + def init_rollout( + self, + *, + metadata: list[Any], + train_world_size: int, + rollout_world_size: int, + ) -> None: + self.engine.init_rollout_process_group( + rollout_rank=self.rank, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + metadata=metadata, + ) + + def finalize(self) -> None: + self.engine.finalize() + + def close(self) -> None: + self.engine.finalize() + disconnect = getattr(self.engine, "_disconnect_peers", None) + if callable(disconnect): + disconnect() + + def send_weights(self, *, phase: int, tensor_mb: int) -> dict[str, Any]: + import torch + + element_count = tensor_mb * 1024 * 1024 // torch.int32.itemsize + base_value = phase * 1000 + self.rank + weights = [ + ( + "dense.weight", + torch.arange( + element_count, + dtype=torch.int32, + device=self.device, + ) + + base_value, + ), + ( + "router.weight", + torch.full( + (4096,), + base_value, + dtype=torch.int32, + device=self.device, + ), + ), + ] + expected = _tensor_summary(weights) + + async def run() -> None: + await self.engine.send_weights(iter(weights)) + + start = time.time() + asyncio.run(run()) + return { + "label": self.label, + "rank": self.rank, + "phase": phase, + "elapsed_s": time.time() - start, + "expected": expected, + } + + def receive_weights(self, *, phase: int) -> dict[str, Any]: + async def run() -> dict[str, Any]: + received: dict[str, Any] = {} + async for batch in self.engine.receive_weight_batches(): + received.update(_tensor_summary(batch)) + return received + + start = time.time() + received = asyncio.run(run()) + return { + "label": self.label, + "rank": self.rank, + "phase": phase, + "elapsed_s": time.time() - start, + "received": received, + } + + +_REMOTE_ENDPOINT_CLASS: Any | None = None + + +def _remote_endpoint_class() -> Any: # pragma: no cover + global _REMOTE_ENDPOINT_CLASS + if _REMOTE_ENDPOINT_CLASS is None: + import ray + + _REMOTE_ENDPOINT_CLASS = ray.remote(ElasticNixlEndpoint) + return _REMOTE_ENDPOINT_CLASS + + +def _new_endpoint( + *, + label: str, + rank: int, + node: dict[str, Any], + backend: str, + bucket_size_bytes: int, + engine_kwargs: dict[str, Any], + device: str, + use_gpu: bool, +) -> Any: # pragma: no cover + return ( + _remote_endpoint_class() + .options(**_actor_options(node, use_gpu=use_gpu)) + .remote( + label=label, + rank=rank, + backend=backend, + bucket_size_bytes=bucket_size_bytes, + engine_kwargs=engine_kwargs, + device=device, + ) + ) + + +def _validate_results( + *, + expected_by_rank: dict[int, dict[str, Any]], + rollout_results: list[dict[str, Any]], +) -> None: + for result in rollout_results: + rank = result["rank"] + expected = expected_by_rank.get(rank) + if expected is None: + raise RuntimeError(f"Received result for unexpected rollout rank {rank}.") + if result["received"] != expected: + raise RuntimeError( + "Rollout result mismatch:\n" + f"rank={rank}\n" + f"expected={json.dumps(expected, sort_keys=True)}\n" + f"actual={json.dumps(result, sort_keys=True)}" + ) + + +def _parse_args() -> argparse.Namespace: # pragma: no cover + parser = argparse.ArgumentParser( + description=( + "Demonstrate NIXL checkpoint-engine peer rebuilds while rollout " + "actors are added and removed." + ) + ) + parser.add_argument("--ray-address", default="auto") + parser.add_argument("--min-nodes", type=int, default=4) + parser.add_argument("--rollout-sequence", default="1,3,2,4,1") + parser.add_argument("--backend", default="nixl") + parser.add_argument("--nixl-backend-name", default="UCX") + parser.add_argument("--device", choices=("cpu", "cuda"), default="cuda") + parser.add_argument("--bucket-mb", type=int, default=64) + parser.add_argument("--tensor-mb", type=int, default=96) + parser.add_argument("--timeout-s", type=int, default=600) + parser.add_argument( + "--ucx-error-handling-mode", + default="none", + help="NIXL UCX backend ucx_error_handling_mode.", + ) + return parser.parse_args() + + +def main() -> None: # pragma: no cover + args = _parse_args() + import ray + + rollout_sequence = _parse_sequence(args.rollout_sequence) + policy_count = max(rollout_sequence) + if policy_count > args.min_nodes: + raise ValueError( + "This demo maps endpoint ranks across the reserved nodes; " + f"max rollout count {policy_count} exceeds --min-nodes={args.min_nodes}." + ) + + ray.init(address=args.ray_address) + nodes = _wait_for_nodes(args.min_nodes, args.timeout_s)[: args.min_nodes] + print( + "ELASTIC_NIXL_DEMO_NODES " + json.dumps([_node_label(node) for node in nodes]), + flush=True, + ) + + use_gpu = args.device == "cuda" + bucket_size_bytes = args.bucket_mb * 1024 * 1024 + engine_kwargs = _build_engine_kwargs(args) + + policy_handles: list[EndpointHandle] = [] + active_rollouts: dict[int, EndpointHandle] = {} + rollout_generations: dict[int, int] = {} + + try: + for rank in range(policy_count): + node = nodes[rank % len(nodes)] + actor = _new_endpoint( + label=f"policy-{rank}", + rank=rank, + node=node, + backend=args.backend, + bucket_size_bytes=bucket_size_bytes, + engine_kwargs=engine_kwargs, + device=args.device, + use_gpu=use_gpu, + ) + policy_handles.append( + EndpointHandle( + rank=rank, + generation=1, + node_id=node["NodeID"], + node_ip=node["NodeManagerAddress"], + actor=actor, + ) + ) + print( + "ELASTIC_NIXL_POLICIES " + + json.dumps( + ray.get([handle.actor.report.remote() for handle in policy_handles]), + sort_keys=True, + ), + flush=True, + ) + + for phase, target_count in enumerate(rollout_sequence, start=1): + added_ranks, removed_ranks = _phase_delta(active_rollouts, target_count) + + for rank in removed_ranks: + handle = active_rollouts.pop(rank) + ray.get(handle.actor.close.remote()) + ray.kill(handle.actor, no_restart=True) + + for rank in added_ranks: + generation = rollout_generations.get(rank, 0) + 1 + rollout_generations[rank] = generation + node = nodes[(rank + 1) % len(nodes)] + actor = _new_endpoint( + label=f"rollout-{rank}-gen-{generation}", + rank=rank, + node=node, + backend=args.backend, + bucket_size_bytes=bucket_size_bytes, + engine_kwargs=engine_kwargs, + device=args.device, + use_gpu=use_gpu, + ) + active_rollouts[rank] = EndpointHandle( + rank=rank, + generation=generation, + node_id=node["NodeID"], + node_ip=node["NodeManagerAddress"], + actor=actor, + ) + + active = [active_rollouts[rank] for rank in sorted(active_rollouts)] + reports = ray.get([handle.actor.report.remote() for handle in active]) + print( + "ELASTIC_NIXL_PHASE " + + json.dumps( + { + "phase": phase, + "target_rollouts": target_count, + "added": added_ranks, + "removed": removed_ranks, + "active": [handle.rank for handle in active], + "reports": reports, + }, + sort_keys=True, + ), + flush=True, + ) + + policy_metadata = ray.get( + [handle.actor.prepare.remote() for handle in policy_handles] + ) + rollout_metadata = ray.get( + [handle.actor.prepare.remote() for handle in active] + ) + metadata = policy_metadata + rollout_metadata + train_world_size = len(policy_handles) + rollout_world_size = len(active) + ray.get( + [ + handle.actor.init_policy.remote( + metadata=metadata, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + ) + for handle in policy_handles + ] + ) + ray.get( + [ + handle.actor.init_rollout.remote( + metadata=metadata, + train_world_size=train_world_size, + rollout_world_size=rollout_world_size, + ) + for handle in active + ] + ) + + receive_refs = [ + handle.actor.receive_weights.remote(phase=phase) for handle in active + ] + send_refs = [ + handle.actor.send_weights.remote( + phase=phase, + tensor_mb=args.tensor_mb, + ) + for handle in policy_handles + ] + send_results = ray.get(send_refs) + rollout_results = ray.get(receive_refs) + expected_by_rank = { + result["rank"]: result["expected"] + for result in send_results + if result["rank"] < rollout_world_size + } + _validate_results( + expected_by_rank=expected_by_rank, + rollout_results=rollout_results, + ) + + ray.get([handle.actor.finalize.remote() for handle in policy_handles]) + ray.get([handle.actor.finalize.remote() for handle in active]) + + print( + "ELASTIC_NIXL_RESULT " + + json.dumps( + { + "phase": phase, + "rollout_count": len(active), + "added": added_ranks, + "removed": removed_ranks, + "send_elapsed_s": [ + round(result["elapsed_s"], 6) for result in send_results + ], + "receive_elapsed_s": [ + round(result["elapsed_s"], 6) for result in rollout_results + ], + "status": "ok", + }, + sort_keys=True, + ), + flush=True, + ) + + finally: + for handle in [*policy_handles, *active_rollouts.values()]: + with contextlib.suppress(Exception): + ray.get(handle.actor.close.remote()) + with contextlib.suppress(Exception): + ray.kill(handle.actor, no_restart=True) + ray.shutdown() + + print("ELASTIC_NIXL_DEMO_COMPLETED", flush=True) + + +if __name__ == "__main__": + main() diff --git a/uv.lock b/uv.lock index ca978d833c3..81ffaf68427 100644 --- a/uv.lock +++ b/uv.lock @@ -3274,9 +3274,8 @@ docs = [ name = "nemo-rl" source = { editable = "." } dependencies = [ - { name = "awscrt", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, - { name = "zstandard", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "accelerate", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "awscrt", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "blobfile", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "colored", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "cuda-bindings", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, @@ -3289,6 +3288,7 @@ dependencies = [ { name = "mooncake-transfer-engine-cuda13", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nccl4py", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "ninja", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "nixl", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "num2words", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "numpy", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "nvidia-cudnn-cu13", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, @@ -3324,6 +3324,7 @@ dependencies = [ { name = "transformers", version = "5.8.1", source = { registry = "https://pypi.org/simple" }, marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'aarch64' and sys_platform == 'linux' and extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang') or (platform_machine == 'x86_64' and sys_platform == 'linux' and extra != 'extra-7-nemo-rl-automodel' and extra != 'extra-7-nemo-rl-sglang') or (platform_machine == 'aarch64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'aarch64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'aarch64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine == 'aarch64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine == 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'x86_64' and extra != 'extra-7-nemo-rl-fsdp' and extra != 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "triton", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, { name = "wandb", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "zstandard", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, ] [package.optional-dependencies] @@ -3444,9 +3445,8 @@ test = [ [package.metadata] requires-dist = [ - { name = "awscrt", specifier = ">=0.35.0" }, - { name = "zstandard" }, { name = "accelerate", specifier = ">=0.26" }, + { name = "awscrt", specifier = ">=0.35.0" }, { name = "blobfile" }, { name = "causal-conv1d", marker = "extra == 'automodel'", git = "https://github.com/Dao-AILab/causal-conv1d?rev=4f6ae4e26ae5fe8af9372f8d312ab25cc4595223" }, { name = "causal-conv1d", marker = "extra == 'fsdp'", git = "https://github.com/Dao-AILab/causal-conv1d?rev=4f6ae4e26ae5fe8af9372f8d312ab25cc4595223" }, @@ -3501,6 +3501,7 @@ requires-dist = [ { name = "nemo-automodel", extras = ["moe"], marker = "extra == 'automodel'", editable = "3rdparty/Automodel-workspace/Automodel" }, { name = "nemo-gym", marker = "extra == 'nemo-gym'", editable = "3rdparty/Gym-workspace/Gym" }, { name = "ninja" }, + { name = "nixl", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')", specifier = "==1.3.0" }, { name = "num2words", specifier = ">=0.5.14" }, { name = "num2words", marker = "extra == 'vllm'", specifier = ">=0.5.14" }, { name = "numpy" }, @@ -3556,6 +3557,7 @@ requires-dist = [ { name = "vllm", marker = "platform_machine == 'x86_64' and sys_platform == 'linux' and extra == 'vllm'", url = "https://github.com/vllm-project/vllm/releases/download/v0.20.0/vllm-0.20.0-cp38-abi3-manylinux_2_35_x86_64.whl" }, { name = "vllm", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'vllm') or (sys_platform != 'linux' and extra == 'vllm')", specifier = "==0.20.0" }, { name = "wandb", specifier = ">=0.28.0" }, + { name = "zstandard" }, ] provides-extras = ["fsdp", "automodel", "vllm", "sglang", "mcore", "modelopt", "nvrx", "nemo-gym"] @@ -3620,6 +3622,44 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/34/6f/5f5a54a1041af945130abdb2b8529cbef0cdcbbf9bcf3f4195378319d29a/ninja-1.13.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b4f2a072db3c0f944c32793e91532d8948d20d9ab83da9c0c7c15b5768072200", size = 581758, upload-time = "2025-08-11T15:10:13.295Z" }, ] +[[package]] +name = "nixl" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nixl-cu12", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "nixl-cu13", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/a8/1e956af6efd181f44d0a94bfc9d883f1b19788467357018fbffe1fbe7a2d/nixl-1.3.0-py3-none-any.whl", hash = "sha256:000e2852040bce7863420c98f2c8ca675dc24066099c746fdbfe2bed3593ec9e", size = 10297, upload-time = "2026-06-15T23:22:51.093Z" }, +] + +[[package]] +name = "nixl-cu12" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/36/f8/9ac816a23a35da5c79ce1e1da80933a013112073a00f4da52eb576a3ee68/nixl_cu12-1.3.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:9a275d532ddb45ecdc5af00f7f47f8945a027d9a1c7a3184d688e151764da2dc", size = 49104015, upload-time = "2026-06-15T23:17:23.474Z" }, + { url = "https://files.pythonhosted.org/packages/cf/1a/2c11c42f71c0acf164c953291ec28297caeab02f571d9a97e7eca2034b3b/nixl_cu12-1.3.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:60eef38ee2b62957de8ba7d3178618b8bb099663bfd111cb19174b5010c7bd77", size = 50558141, upload-time = "2026-06-15T23:15:00.616Z" }, +] + +[[package]] +name = "nixl-cu13" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, + { name = "torch", marker = "(platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine != 'aarch64' and platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-fsdp') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-mcore') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-automodel' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-fsdp' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-sglang') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-mcore' and extra == 'extra-7-nemo-rl-vllm') or (sys_platform != 'linux' and extra == 'extra-7-nemo-rl-sglang' and extra == 'extra-7-nemo-rl-vllm')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/ea/ff/9f462e977a40128c088bdc698ab855635caa5d7511fca245365ebf5d5e45/nixl_cu13-1.3.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:7c42f8546cfbe4595669212d4bbf596590f61494f0ae1a3bfbd3fb4fe4abf84f", size = 43678469, upload-time = "2026-06-15T23:21:58.728Z" }, + { url = "https://files.pythonhosted.org/packages/67/96/d01c466193e88fcc4afeb2dc29ff35512b4af497db7a2e76e95f3f75b821/nixl_cu13-1.3.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:2363d3922402f7f3b7302254c3b44cc71520e5ad6a9dc43898fa59a6ee0620a3", size = 45134021, upload-time = "2026-06-15T23:19:50.763Z" }, +] + [[package]] name = "nodeenv" version = "1.10.0" @@ -7158,6 +7198,14 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/bf/2e/0b49f7e4e53817cfb09a0f6585012b782dfe0b666e8abefcb4fac0570606/z3_solver-4.15.4.0-py3-none-manylinux_2_34_aarch64.whl", hash = "sha256:62c7e9cbdd711932301f29919ad9158de9b2f58b4d281dd259bbcd0a2f408ba1", size = 27226534, upload-time = "2025-10-29T18:11:55.59Z" }, ] +[[package]] +name = "zipp" +version = "3.23.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/30/21/093488dfc7cc8964ded15ab726fad40f25fd3d788fd741cc1c5a17d78ee8/zipp-3.23.1.tar.gz", hash = "sha256:32120e378d32cd9714ad503c1d024619063ec28aad2248dc6672ad13edfa5110", size = 25965, upload-time = "2026-04-13T23:21:46.6Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/08/8a/0861bec20485572fbddf3dfba2910e38fe249796cb73ecdeb74e07eeb8d3/zipp-3.23.1-py3-none-any.whl", hash = "sha256:0b3596c50a5c700c9cb40ba8d86d9f2cc4807e9bedb06bcdf7fac85633e444dc", size = 10378, upload-time = "2026-04-13T23:21:45.386Z" }, +] [[package]] name = "zstandard" @@ -7172,11 +7220,3 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" }, { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" }, ] -[[package]] -name = "zipp" -version = "3.23.1" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/30/21/093488dfc7cc8964ded15ab726fad40f25fd3d788fd741cc1c5a17d78ee8/zipp-3.23.1.tar.gz", hash = "sha256:32120e378d32cd9714ad503c1d024619063ec28aad2248dc6672ad13edfa5110", size = 25965, upload-time = "2026-04-13T23:21:46.6Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/08/8a/0861bec20485572fbddf3dfba2910e38fe249796cb73ecdeb74e07eeb8d3/zipp-3.23.1-py3-none-any.whl", hash = "sha256:0b3596c50a5c700c9cb40ba8d86d9f2cc4807e9bedb06bcdf7fac85633e444dc", size = 10378, upload-time = "2026-04-13T23:21:45.386Z" }, -]