Update defaults in FSDPStrategy for limit_all_gathers and backward_pr…

…efetch (pytorch#457)

Summary:
Pull Request resolved: pytorch#457

FSDP default recently changed: https://github.com/pytorch/pytorch/pull/104900/files, so updating here as well

TODO: is there a better way to keep these dataclasses in sync with pytorch?

Reviewed By: daniellepintz

Differential Revision: D47449311

fbshipit-source-id: f2cfc152fa34dc5bf695ad941d72a3a2e0f4dfe7

torchtnt/utils/prepare_module.py

@@ -67,11 +67,11 @@ class FSDPStrategy(Strategy):
     sharding_strategy: Optional[ShardingStrategy] = None
     cpu_offload: Optional[CPUOffload] = None
     auto_wrap_policy: Optional[Callable[[torch.nn.Module, bool, int], bool]] = None
-    backward_prefetch: Optional[BackwardPrefetch] = None
+    backward_prefetch: Optional[BackwardPrefetch] = BackwardPrefetch.BACKWARD_PRE
     ignored_modules: Optional[Iterable[torch.nn.Module]] = None
     sync_module_states: bool = False
     forward_prefetch: bool = False
-    limit_all_gathers: bool = False
+    limit_all_gathers: bool = True
     use_orig_params: bool = False
 
     # FSDP set_state_dict_type params: https://pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel.set_state_dict_type