Merge pull request #648 from allenai/shanea/default-fsdp-strategy

Default to FSDP strategy
allenai · Jul 15, 2024 · 26c2d53 · 26c2d53
2 parents 65f1fff + 75d45c6
commit 26c2d53
Show file tree

Hide file tree

Showing 2 changed files with 5 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+### Changed
+
+- Changed default distributed training strategy from single-GPU to FSDP
+
 ## [v0.4.0](https://github.com/allenai/OLMo/releases/tag/v0.4.0) - 2024-07-11
 
 ### Added

diff --git a/olmo/config.py b/olmo/config.py
@@ -1104,7 +1104,7 @@ class TrainConfig(BaseConfig):
     Settings for compiling the model with ``torch.compile()``.
     """
 
-    distributed_strategy: Optional[DistributedStrategy] = None
+    distributed_strategy: Optional[DistributedStrategy] = DistributedStrategy.fsdp
     """
     Distributed strategy for OLMo model (eg. single GPU, DDP, FSDP).
     """