feat: add checkpoint-engine refit interface and integrate NIXL

[HollowMan6]

What does this PR do?

Adds NIXL checkpoint-engine refit for transferring policy weights to
non-colocated vLLM generation workers.

The implementation:

• Integrates NIXL with the existing refit_transport and weight-synchronizer
  lifecycle.
• Supports DTensor, DTensor v2, and Megatron policy backends with vLLM
  generation.
• Supports CUDA UCX/RDMA transfers and host-pinned CPU transfers.
• Sizes each of the two transfer buffers from a configurable fraction of total
  GPU memory.
• Keeps communicators and buffers across refits by default, with optional buffer
  release after each refit.
• Provides a readable full-weight loading path and an optional
  destination-layout-aware sharded-expert path.
• Validates unsupported topology, quantization, and expert-layout combinations
  before transfer.
• Makes NIXL available in isolated policy and vLLM actor environments on Linux
  x86-64 and ARM64.

Limitations

• Only non-colocated vLLM generation is supported.
• shard_expert_weights: true currently supports unquantized Triton MoE
  storage.
• Sharded FP8/MXFP8 expert refit is not supported. FP8 models can use
  full-weight refit with shard_expert_weights: false.
• Dynamic expert load balancing and redundant expert placement are not
  supported by the sharded-expert path.
• Support for other generation backends is tracked in feat: extend checkpoint-engine refit to SGLang (and Megatron) generation backends #3288.

Issues

None.

Usage

Enable NIXL for non-colocated vLLM generation:

policy:
  generation:
    backend: vllm
    colocated:
      enabled: false
    refit_transport: nixl
    refit_cfg:
      nixl:
        # Fraction of total GPU memory allocated to each transfer buffer.
        # NIXL uses two buffers, so 0.05 reserves up to 10% in total.
        update_weights_bucket_memory_ratio: 0.05
        device: cuda
        backend_name: UCX
        release_after_refit: false
        shard_expert_weights: false

For MoE models, enable destination-local expert transfer with:

policy:
  generation:
    refit_cfg:
      nixl:
        shard_expert_weights: true

NIC and rail selection can be configured through backend_init_params.
Site-specific device names are intentionally not included in the minimal
configuration.

See:

• docs/guides/refit.md
• docs/guides/checkpoint-engine-refit.md
• docs/design-docs/checkpoint-engines.md

Performance

DeepSeek-V3 BF16 was benchmarked with 32 Megatron policy nodes and four vLLM
rollout nodes over eight RDMA rails:

Comparison	NIXL	NCCL	Improvement
Dedicated asynchronous refit	9.25 s	14.21 s	1.54x
Synchronized verifier	10.92 s	15.52 s	1.42x

Within the same NIXL build, destination-local sharded-expert transfer reduced
refit time from 36.81 s to 10.92 s, a 3.37x improvement.
tools/refit_verifier.py reported identical aggregate outputs and log
probabilities for the full- and sharded-expert paths.

Testing

• Added unit coverage for NIXL bucket splitting, double-buffering, transfer
  lifecycle, async receive/load dispatch, and sharded expert placement.
• Added a fast non-colocated NIXL GRPO functional smoke test.
• Ran a two-step end-to-end GRPO refit with fresh isolated Ray actor
  environments.
• Verified both actor environments import NIXL 1.3.0 and complete repeated
  refits.
• Verified DeepSeek-V3 full- and sharded-expert refit with
  tools/refit_verifier.py.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

The default configuration retains NIXL transfer buffers and peer connections
across refits for throughput. Set release_after_refit: true when reclaiming
the transfer-buffer memory between refits is more important than avoiding
buffer registration and allocation overhead.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[HollowMan6]

/ok to test 705af30

[ZhiyuLi-Nvidia]

Awesome @HollowMan6. Generally looks great to me!

[HollowMan6]

/ok to test 4d89446

[terrykong]

Re-review at head 4d89446b3 (rebased on top of the now-merged delta-refit #2444). Prior rounds were thorough — every earlier "fixed" claim landed, the lint failure I flagged last round is fixed, and the NIXL double-buffer handshake, bucket split/merge, and sharded w13/w2 placement were independently verified against vLLM 0.20.0 and check out.

The main new point from the rebase is structural, not a bug: delta-refit and this PR both add a non-colocated vLLM refit backend, but they're selected and dispatched three different ways (refit_transport string enum vs checkpoint_engine.enabled block vs the legacy inline path), with no single selector and no mutual-exclusion guard. The lead inline comment lays this out — aligning this PR with delta-refit's weight_synchronizer-at-setup pattern (routed through the existing-but-unused create_weight_synchronizer factory) collapses the dispatch to one path and makes the colocated guard actually run. A single docs/guides/refit.md explaining how to pick among the refit methods is also missing.

Minor / non-blocking: the PR description still shows the old update_weights_bucket_megabytes / cleanup_after_load fields (now update_weights_bucket_memory_ratio / release_after_refit); worth refreshing and surfacing the guide's NIXL-vs-NCCL benchmark table.

Generated by Claude Code

[HollowMan6]

/ok to test 0916403

[HollowMan6]

/ok to test 64c42b6