Skip to content

docs(distributed): review frozen multimodal FSDP guidance #3272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akoumpa merged 2 commits into from
Jul 30, 2026
Merged

docs(distributed): review frozen multimodal FSDP guidance #3272

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3f7979a
docs(distributed): mark frozen multimodal sections for review
yuhezhang-ai Jul 28, 2026
4560ee1
docs(distributed): apply technical publications review
yuhezhang-ai Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions docs/breaking-changes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ position: 6

### Frozen Multimodal FSDP2 Modules Default to Root Sharding

Fully frozen vision/audio towers and multimodal projectors now default to the `root` FSDP2 policy. Previously, dense
Fully frozen vision and audio towers and multimodal projectors now default to the `root` FSDP2 policy. Previously, dense
frozen vision towers commonly used per-layer FSDP units. Root ownership keeps the collective sequence aligned when
some ranks execute a modality branch and other ranks skip it, but it can increase peak unshard memory.

Expand All @@ -27,7 +27,7 @@ policies and the `wrap_outer_model` constraint.

### FSDP2 Default `reduce_dtype` Is Now `float32`

The default [`MixedPrecisionPolicy`](https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html) built by `FSDP2Config` now uses `reduce_dtype=torch.float32` instead of `torch.bfloat16`. Forward/backward compute still uses `param_dtype=torch.bfloat16`, but gradient reduction now accumulates in fp32 to reduce communication-rounding error at larger data-parallel world sizes.
The default [`MixedPrecisionPolicy`](https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html) built by `FSDP2Config` now uses `reduce_dtype=torch.float32` instead of `torch.bfloat16`. Forward/backward compute still uses `param_dtype=torch.bfloat16`, but gradient reduction now accumulates in FP32 to reduce communication-rounding error at larger data-parallel world sizes.

To restore the previous behavior, override the policy explicitly:

Expand Down Expand Up @@ -97,15 +97,15 @@ entirely within the YAML config file rather than through CLI arguments.

| Launcher | YAML section |
|---|---|
| Kubernetes (via SkyPilot) | `skypilot:` with `cloud: kubernetes` |
| Kubernetes (through SkyPilot) | `skypilot:` with `cloud: kubernetes` |
| NeMo Run | `nemo_run:` |

If neither section is present, the job runs locally (interactive mode).

### SLURM: Script-Based Submission
### Slurm: Script-Based Submission

The `slurm:` YAML section and all related fields have been removed. SLURM
jobs are now submitted with `sbatch` directly, using a self-contained sbatch
The `slurm:` YAML section and all related fields have been removed. Slurm
jobs are now submitted with `sbatch` directly, using a self-contained `sbatch`
script. Copy the reference template and adapt it to your cluster:

```bash
Expand All @@ -117,7 +117,7 @@ sbatch my_cluster.sub
The script runs `torchrun -m nemo_automodel.cli.app` on each node, which
detects the distributed environment and executes the recipe in-process.
All cluster-specific configuration lives in the sbatch script where you can
see and edit it directly.
edit it directly.

### CLI Install Extra

Expand All @@ -136,7 +136,7 @@ to be installed separately.

### Media Dependencies Are Opt-In

The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utils, and
The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utilities, and
imageio-ffmpeg) are now opt-in extras, installed neither by default nor in the
Docker container.

Expand All @@ -149,15 +149,15 @@ uv pip install "nemo-automodel[diffusion,diffusion-media]" # diffusion preproce

Two consequences when upgrading:

- `[vlm]` alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs — add `vlm-media`.
- `[all]` no longer includes the media extras — add them with `uv pip install "nemo-automodel[media]"` in the activated environment.
- `[vlm]` alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs. Add `vlm-media`.
- `[all]` no longer includes the media extras. Add them with `uv pip install "nemo-automodel[media]"` in the activated environment.

See the [Installation Guide](/get-started/installation#media-extras-video--image-decode) for details.

### CLI Module Lives Inside the Package

The CLI entry-point lives at `nemo_automodel/cli/app.py` and is registered as
the `automodel` / `am` console entry-points. A thin convenience wrapper
The CLI entry point lives at `nemo_automodel/cli/app.py` and is registered as
the `automodel` or `am` console entry points. A thin convenience wrapper
(`app.py`) at the repository root is available for running from a source
checkout but is **not** installed as part of the package.

Expand Down
Loading