Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support FSDP in Lite #14898

Closed
awaelchli opened this issue Sep 26, 2022 · 0 comments · Fixed by #14967
Closed

Support FSDP in Lite #14898

awaelchli opened this issue Sep 26, 2022 · 0 comments · Fixed by #14967
Assignees
Labels
fabric lightning.fabric.Fabric feature Is an improvement or enhancement
Milestone

Comments

@awaelchli
Copy link
Contributor

🚀 Feature

Motivation

Native PyTorch FSDP is a powerful alternative to regular DDP.

Pitch

Add the strategy to Lite. Build it using the existing implementation in Trainer.


If you enjoy Lightning, check out our other projects! ⚡

  • Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

  • Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

  • Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

  • Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

  • Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.

@awaelchli awaelchli added feature Is an improvement or enhancement fabric lightning.fabric.Fabric labels Sep 26, 2022
@awaelchli awaelchli added this to the pl:future milestone Sep 26, 2022
@awaelchli awaelchli self-assigned this Oct 10, 2022
@awaelchli awaelchli modified the milestones: future, v1.9 Oct 16, 2022
@awaelchli awaelchli moved this to In Progress in Frameworks Planning Oct 16, 2022
Repository owner moved this from In Progress to Done in Frameworks Planning Nov 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fabric lightning.fabric.Fabric feature Is an improvement or enhancement
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant