Smartsim data loader for PyTorch #106
Assignees
Labels
area: third-party
Issues related to Issues related to dependencies and third-part and third-party package integrations
type: feature
Issues that include feature request or feature idea
Comments
al-rigazzi
added
area: third-party
2 tasks
al-rigazzi
added a commit
that referenced
this issue
Feb 3, 2022
This PR adds data loaders which can be used with TensorFlow and PyTorch to train (or fit) a model. The loaders get the data from the Orchestrator and can be used when training in a distributed fashion (e.g with Horovod). [ committed by @al-rigazzi ] [ reviewed by @Spartee and @MattToast ]
al-rigazzi
added a commit
to al-rigazzi/SmartSim
that referenced
this issue
Feb 4, 2022
This PR adds data loaders which can be used with TensorFlow and PyTorch to train (or fit) a model. The loaders get the data from the Orchestrator and can be used when training in a distributed fashion (e.g with Horovod). [ committed by @al-rigazzi ] [ reviewed by @Spartee and @MattToast ]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
To enable users to use tensors stored in the DB to perform training, we want to offer ready-to-use PyTorch data loaders. This is one of the two stories which will implement what was investigated in #49
Justification
Users who want to implement training services using SmartSim will be able to simply include
from import smartsim.exp.data import SmartPyTorchDataLoaderand immediately start training.Implementation Strategy
A simple one, acting as base class will simply download the tensor from the DB and use it to train a NN, basically optimizing
Model(x)=y. Inheriting from the base class, a more sophisticated one will keep checking for availability of new data.The text was updated successfully, but these errors were encountered: