diff --git a/README.md b/README.md index afba315752..d550588b60 100644 --- a/README.md +++ b/README.md @@ -557,6 +557,41 @@ classifier = LinearClassifier(128, 10) When you reach the limits of the flexibility provided by Flash, then seamlessly transition to PyTorch Lightning which gives you the most flexibility because it is simply organized PyTorch. +## Visualization + +Predictions from image and video tasks can be visualized through an [integration with FiftyOne](https://lightning-flash.readthedocs.io/en/latest/integrations/fiftyone.html), allowing you to better understand and analyze how your model is performing. + +```python +from flash.core.data.utils import download_data +from flash.core.integrations.fiftyone import visualize +from flash.image import ObjectDetector +from flash.image.detection.serialization import FiftyOneDetectionLabels + +# 1. Download the data +# Dataset Credit: https://www.kaggle.com/ultralytics/coco128 +download_data( + "https://github.com/zhiqwang/yolov5-rt-stack/releases/download/v0.3.0/coco128.zip", + "data/", +) + +# 2. Load the model from a checkpoint and use the FiftyOne serializer +model = ObjectDetector.load_from_checkpoint( + "https://flash-weights.s3.amazonaws.com/object_detection_model.pt" +) +model.serializer = FiftyOneDetectionLabels() + +# 3. Detect the object on the images +filepaths = [ + "data/coco128/images/train2017/000000000025.jpg", + "data/coco128/images/train2017/000000000520.jpg", + "data/coco128/images/train2017/000000000532.jpg", +] +predictions = model.predict(filepaths) + +# 4. Visualize predictions in FiftyOne App +session = visualize(predictions, filepaths=filepaths) +``` + ## Contribute! The lightning + Flash team is hard at work building more tasks for common deep-learning use cases. But we're looking for incredible contributors like you to submit new tasks! diff --git a/docs/source/index.rst b/docs/source/index.rst index 29ac911fda..fdca8ed270 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -70,7 +70,6 @@ Lightning Flash integrations/fiftyone - .. toctree:: :maxdepth: 1 :caption: Contributing a Task diff --git a/docs/source/integrations/fiftyone.rst b/docs/source/integrations/fiftyone.rst index d12392e5ec..25aa342727 100644 --- a/docs/source/integrations/fiftyone.rst +++ b/docs/source/integrations/fiftyone.rst @@ -27,14 +27,13 @@ are supported! Installation ************ -In order to utilize this integration with FiftyOne, you will need to -:ref:`install the tool`: +In order to utilize this integration, you will need to +:ref:`install FiftyOne `: .. code:: shell pip install fiftyone - ***************************** Visualizing Flash predictions ***************************** @@ -52,14 +51,30 @@ one of the following serializers: The :func:`~flash.core.integrations.fiftyone.visualize` function then lets you visualize your predictions in the :ref:`FiftyOne App `. This function accepts a list of -dictionaries containing :ref:`FiftyOne Label` objects -and filepaths which is the exact output of the FiftyOne serializers when the flag -``return_filepath=True`` is specified. +dictionaries containing :ref:`FiftyOne Label ` objects +and filepaths, which is exactly the output of the FiftyOne serializers when the +``return_filepath=True`` option is specified. .. literalinclude:: ../../../flash_examples/integrations/fiftyone/image_classification.py :language: python :lines: 14- +The :func:`~flash.core.integrations.fiftyone.visualize` function can be used in +all of the following environments: + +- **Local Python shell**: The App will launch in a new tab in your default + web browser +- **Remote Python shell**: Pass the ``remote=True`` option and then follow + the instructions printed to your remote shell to open the App in your + browser on your local machine +- **Jupyter notebook**: The App will launch in the output of your current + cell +- **Google Colab**: The App will launch in the output of your current cell +- **Python script**: Pass the ``wait=True`` option to block execution of your + script until the App is closed + +See :ref:`this page ` for more information about +using the FiftyOne App in different environments. *********************** Using FiftyOne datasets @@ -73,7 +88,7 @@ ground truth annotations. This allows you to perform more complex analysis with The :meth:`~flash.core.data.data_module.DataModule.from_fiftyone` -method allows you to load your FiftyOne Datasets directly into a +method allows you to load your FiftyOne datasets directly into a :class:`~flash.core.data.data_module.DataModule` to be used for training, testing, or inference. @@ -81,7 +96,6 @@ testing, or inference. :language: python :lines: 14- - ********************** Visualizing embeddings ********************** @@ -130,6 +144,7 @@ FiftyOneSegmentationLabels .. autoclass:: flash.image.segmentation.serialization.FiftyOneSegmentationLabels :members: + :noindex: .. _fiftyone_detection_labels: diff --git a/flash/core/integrations/fiftyone/utils.py b/flash/core/integrations/fiftyone/utils.py index 5498a801ed..9b34249cbd 100644 --- a/flash/core/integrations/fiftyone/utils.py +++ b/flash/core/integrations/fiftyone/utils.py @@ -20,26 +20,47 @@ def visualize( - labels: Union[List[Label], List[Dict[str, Label]]], + predictions: Union[List[Label], List[Dict[str, Label]]], filepaths: Optional[List[str]] = None, - wait: Optional[bool] = True, label_field: Optional[str] = "predictions", + wait: Optional[bool] = False, **kwargs ) -> Optional[Session]: - """Use the result of a FiftyOne serializer to visualize predictions in the - FiftyOne App. + """Visualizes predictions from a model with a FiftyOne Serializer in the + :ref:`FiftyOne App `. + + This method can be used in all of the following environments: + + - **Local Python shell**: The App will launch in a new tab in your + default web browser. + - **Remote Python shell**: Pass the ``remote=True`` option to this method + and then follow the instructions printed to your remote shell to open + the App in your browser on your local machine. + - **Jupyter notebook**: The App will launch in the output of your current + cell. + - **Google Colab**: The App will launch in the output of your current + cell. + - **Python script**: Pass the ``wait=True`` option to block execution of + your script until the App is closed. + + See :ref:`this page ` for more information about + using the FiftyOne App in different environments. Args: - labels: Either a list of FiftyOne labels that will be applied to the - corresponding filepaths provided with through `filepath` or - `datamodule`. Or a list of dictionaries containing image/video - filepaths and corresponding FiftyOne labels. + predictions: Can be either a list of FiftyOne labels that will be + matched with the corresponding ``filepaths``, or a list of + dictionaries with "filepath" and "predictions" keys that contains + the filepaths and predictions. filepaths: A list of filepaths to images or videos corresponding to the - provided `labels`. - wait: A boolean determining whether to launch the FiftyOne session and - wait until the session is closed or whether to return immediately. - label_field: The string of the label field in the FiftyOne dataset - containing predictions + provided ``predictions``. + label_field: The name of the label field in which to store the + predictions in the FiftyOne dataset. + wait: Whether to block execution until the FiftyOne App is closed. + **kwargs: Optional keyword arguments for + :meth:`fiftyone:fiftyone.core.session.launch_app`. + + Returns: + a :class:`fiftyone:fiftyone.core.session.Session` """ if not _FIFTYONE_AVAILABLE: raise ModuleNotFoundError("Please, `pip install fiftyone`.") @@ -47,24 +68,24 @@ def visualize( return None # Flatten list if batches were used - if all(isinstance(fl, list) for fl in labels): - labels = list(chain.from_iterable(labels)) + if all(isinstance(fl, list) for fl in predictions): + predictions = list(chain.from_iterable(predictions)) - if all(isinstance(fl, dict) for fl in labels): - filepaths = [lab["filepath"] for lab in labels] - labels = [lab["predictions"] for lab in labels] + if all(isinstance(fl, dict) for fl in predictions): + filepaths = [lab["filepath"] for lab in predictions] + labels = [lab["predictions"] for lab in predictions] + else: + labels = predictions if filepaths is None: raise ValueError("The `filepaths` argument is required if filepaths are not provided in `labels`.") dataset = fo.Dataset() if filepaths: - dataset.add_labeled_images( - list(zip(filepaths, labels)), - LabeledImageTupleSampleParser(), - label_field=label_field, - ) + dataset.add_samples([fo.Sample(filepath=f, **{label_field: l}) for f, l in zip(filepaths, labels)]) + session = fo.launch_app(dataset, **kwargs) if wait: session.wait() + return session diff --git a/flash/image/segmentation/serialization.py b/flash/image/segmentation/serialization.py index 574be138d9..4c92fb9287 100644 --- a/flash/image/segmentation/serialization.py +++ b/flash/image/segmentation/serialization.py @@ -39,15 +39,16 @@ class SegmentationLabels(Serializer): + """A :class:`.Serializer` which converts the model outputs to the label of + the argmax classification per pixel in the image for semantic segmentation + tasks. - def __init__(self, labels_map: Optional[Dict[int, Tuple[int, int, int]]] = None, visualize: bool = False): - """A :class:`.Serializer` which converts the model outputs to the label of the argmax classification - per pixel in the image for semantic segmentation tasks. + Args: + labels_map: A dictionary that map the labels ids to pixel intensities. + visualize: Wether to visualize the image labels. + """ - Args: - labels_map: A dictionary that map the labels ids to pixel intensities. - visualize: Wether to visualize the image labels. - """ + def __init__(self, labels_map: Optional[Dict[int, Tuple[int, int, int]]] = None, visualize: bool = False): super().__init__() self.labels_map = labels_map self.visualize = visualize @@ -89,6 +90,16 @@ def serialize(self, sample: Dict[str, torch.Tensor]) -> torch.Tensor: class FiftyOneSegmentationLabels(SegmentationLabels): + """A :class:`.Serializer` which converts the model outputs to FiftyOne + segmentation format. + + Args: + labels_map: A dictionary that map the labels ids to pixel intensities. + visualize: whether to visualize the image labels. + return_filepath: Boolean determining whether to return a dict + containing filepath and FiftyOne labels (True) or only a list of + FiftyOne labels (False). + """ def __init__( self, @@ -96,15 +107,6 @@ def __init__( visualize: bool = False, return_filepath: bool = False, ): - """A :class:`.Serializer` which converts the model outputs to FiftyOne segmentation format. - - Args: - labels_map: A dictionary that map the labels ids to pixel intensities. - visualize: Wether to visualize the image labels. - return_filepath: Boolean determining whether to return a dict - containing filepath and FiftyOne labels (True) or only a - list of FiftyOne labels (False) - """ if not _FIFTYONE_AVAILABLE: raise ModuleNotFoundError("Please, run `pip install fiftyone`.") diff --git a/flash_examples/integrations/fiftyone/image_classification.py b/flash_examples/integrations/fiftyone/image_classification.py index 2b9e13717b..50683e840c 100644 --- a/flash_examples/integrations/fiftyone/image_classification.py +++ b/flash_examples/integrations/fiftyone/image_classification.py @@ -51,11 +51,10 @@ # 4 Predict from checkpoint model = ImageClassifier.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/image_classification_model.pt") -model.serializer = FiftyOneLabels(return_filepath=True) +model.serializer = FiftyOneLabels(return_filepath=True) # output FiftyOne format predictions = trainer.predict(model, datamodule=datamodule) - predictions = list(chain.from_iterable(predictions)) # flatten batches -# 5. Visualize predictions in FiftyOne -# Note: this blocks until the FiftyOne App is closed +# 5 Visualize predictions in FiftyOne App +# Optional: pass `wait=True` to block execution until App is closed session = visualize(predictions) diff --git a/flash_examples/integrations/fiftyone/image_classification_fiftyone_datasets.py b/flash_examples/integrations/fiftyone/image_classification_fiftyone_datasets.py index d7bc4cb72a..ec622cb5a9 100644 --- a/flash_examples/integrations/fiftyone/image_classification_fiftyone_datasets.py +++ b/flash_examples/integrations/fiftyone/image_classification_fiftyone_datasets.py @@ -39,14 +39,14 @@ dataset_type=fo.types.ImageClassificationDirectoryTree, ) -# 3 Load FiftyOne datasets +# 3 Load data into Flash datamodule = ImageClassificationData.from_fiftyone( train_dataset=train_dataset, val_dataset=val_dataset, test_dataset=test_dataset, ) -# 4 Fine tune a model +# 4 Fine tune model model = ImageClassifier( backbone="resnet18", num_classes=datamodule.num_classes, @@ -66,28 +66,22 @@ # 5 Predict from checkpoint on data with ground truth model = ImageClassifier.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/image_classification_model.pt") -model.serializer = FiftyOneLabels(return_filepath=False) +model.serializer = FiftyOneLabels(return_filepath=False) # output FiftyOne format datamodule = ImageClassificationData.from_fiftyone(predict_dataset=test_dataset) predictions = trainer.predict(model, datamodule=datamodule) - predictions = list(chain.from_iterable(predictions)) # flatten batches # 6 Add predictions to dataset test_dataset.set_values("predictions", predictions) -# 7 Visualize labels in the App -session = fo.launch_app(test_dataset) - -# 8 Evaluate your model -results = test_dataset.evaluate_classifications( - "predictions", - gt_field="ground_truth", - eval_key="eval", -) +# 7 Evaluate your model +results = test_dataset.evaluate_classifications("predictions", gt_field="ground_truth", eval_key="eval") results.print_report() plot = results.plot_confusion_matrix() plot.show() -# Only when running this in a script -# Block until the FiftyOne App is closed +# 8 Visualize results in the App +session = fo.launch_app(test_dataset) + +# Optional: block execution until App is closed session.wait() diff --git a/flash_examples/integrations/fiftyone/image_embedding.py b/flash_examples/integrations/fiftyone/image_embedding.py index 71b9ef68e3..7829885d94 100644 --- a/flash_examples/integrations/fiftyone/image_embedding.py +++ b/flash_examples/integrations/fiftyone/image_embedding.py @@ -29,7 +29,7 @@ ) # 3 Load model -embedder = ImageEmbedder(backbone="swav-imagenet", embedding_dim=128) +embedder = ImageEmbedder(backbone="resnet101", embedding_dim=128) # 4 Generate embeddings filepaths = dataset.values("filepath") @@ -37,12 +37,9 @@ # 5 Visualize in FiftyOne App results = fob.compute_visualization(dataset, embeddings=embeddings) - session = fo.launch_app(dataset) - plot = results.visualize(labels="ground_truth.label") plot.show() -# Only when running this in a script -# Block until the FiftyOne App is closed +# Optional: block execution until App is closed session.wait()