Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for applying Lightning Flash models to datasets #1059

Merged
merged 64 commits into from
Jun 21, 2021

Conversation

ehofesmann
Copy link
Member

@ehofesmann ehofesmann commented May 29, 2021

Adds support for applying Lightning Flash models to datasets via SampleCollection.apply_model() and SampleCollection.compute_embeddings().

This integration requires flash==0.4.0 or later.

This branch doubles as the release branch for fiftyone==0.10.0.

Predictions example:

from flash.image import ObjectDetector
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart").exclude_fields("predictions").clone()

model = ObjectDetector.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/object_detection_model.pt")

view = dataset.take(5)
view.apply_model(model, label_field="predictions")

session = fo.launch_app(view=view)

Embeddings example:

from flash.image import ImageEmbedder
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

model = ImageEmbedder(backbone="resnet101", embedding_dim=128)

view = dataset.take(5)
embeddings = view.compute_embeddings(model)

print(embeddings.shape)

See Lightning-Universe/lightning-flash#360 for many more examples!

@ehofesmann ehofesmann self-assigned this May 29, 2021
@ehofesmann ehofesmann added core Issues related to Core features feature Work on a feature request labels May 29, 2021
fiftyone/core/models.py Outdated Show resolved Hide resolved
fiftyone/core/models.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
fiftyone/utils/flash.py Outdated Show resolved Hide resolved
@brimoor brimoor changed the title PyTorch Lightning Flash video classification model integration PyTorch Lightning Flash integration Jun 3, 2021
@brimoor brimoor changed the title PyTorch Lightning Flash integration Adding support for applying PyTorch Lightning Flash models to datasets Jun 3, 2021
@brimoor brimoor marked this pull request as ready for review June 3, 2021 19:51
@brimoor brimoor requested review from brimoor and benjaminpkane June 21, 2021 15:38
@benjaminpkane
Copy link
Contributor

Released

@benjaminpkane benjaminpkane merged commit 076eccb into develop Jun 21, 2021
@benjaminpkane benjaminpkane deleted the flash-video branch June 21, 2021 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Issues related to Core features feature Work on a feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants