[FR] Way to Pre-render thumbnails of the patches you see in patch view

[quitmeyer]

Proposal Summary

maybe this already exists, but I want to be able to pre-render out the crops of the views you see in patch view, and have those load at run time

Motivation

I have a dataset that i have been having major performance issues with. The full size images at 10,000 pixels wide and i have about 500 of them. Each image can have like 1-30 detections that are generally like 200 pixels and smaller. When i load them up in the fiftyone app, stuff gets real slow (to the point of unusability) and often crashes. I have been battling with this for a couple months now, and one solution i had was to generate thumbnails for all the samples with some code like this:

 samples_to_process = []

    for sample in dataset:
        filename = os.path.basename(sample.filepath)
        thumbnail_path = f"{output_dir}/{filename}"
        sample["thumbnail_path"]=thumbnail_path
        sample.save()
        if not os.path.exists(thumbnail_path):
            samples_to_process.append(sample)
        else:
          sample["thumbnail_path"]=thumbnail_path

    if samples_to_process:
            # Create a new dataset with the samples to process
      dataset_to_process = fo.Dataset()
      dataset_to_process.add_samples(samples_to_process)
      print("making extra thumbnails")
      print(samples_to_process)
      foui.transform_images(
          dataset_to_process,
          size=target_size,
          output_field="thumbnail_path",
          output_dir=output_dir,
      )
    dataset.save()

now i get two options when using the app! I have speed, but pixelation, or full res but un-usable.

patches taken from shrunken samples (runs fast! can't ID them too pixely)

patches from full size images (goes super slow and often crashes everything)

Instead it would make more sense to be able to have the patch view load pre-rendered thumbnails of each patch (not a thumbnail of the FULL sample, but rather the patches)

Since the patches are generally like 200 pixels and smaller the performance would be great, and this performance wouldn't lower the cropped image quality.

maybe this is something one can already do, but i don't really know how?

This "tips and tricks" thing suggests something related, but i don't really understand it.
https://voxel51.com/blog/fiftyone-computer-vision-tips-and-tricks-may-19-2023/

What areas of FiftyOne does this feature affect?

• [X ] App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently
• Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• [x ] No. I cannot contribute this feature at this time, i dont know how to code a fiftyone

[quitmeyer]

I could, it seems, export the patches to a new dataset that has multiple media fields that point to a rendered patch and the original media file. This seems like a bit of a hack that can get kinda messy though, especially when there already exists the "patch" feature in fiftyone

[quitmeyer]

FYI, thanks to a person in the slack's help, i figured a hack to make the performance go fantastic by doing what i was looking for (pre-rendering patches), but you basically convert your "proper" dataset (with detections and patches) into a different dataset that sort of loses that patch data, but runs very fast

https://github.com/Digital-Naturalism-Laboratories/Mothbox/blob/main/AI/Mothbot_ConvertAnyLabelsto51_51proper.py

[quitmeyer]

FYI, thanks to a person in the slack's help, i figured a hack to make the performance go fantastic by doing what i was looking for (pre-rendering patches), but you basically convert your "proper" dataset (with detections and patches) into a different dataset that sort of loses that patch data, but runs very fast

https://github.com/Digital-Naturalism-Laboratories/Mothbox/blob/main/AI/Mothbot_ConvertAnyLabelsto51_51proper.py

The downside to this method is that if one of the thumbnails changes in a dataset, you pretty much need to re-render all the thumbnail patches over again. So still seems like there should be a much better option