[BUG] evaluate_detections() is skipping detections

[AntoninDuvalPreligens]

Describe the problem

Hi ! Thanks for this awesome project !

I've been using 51 for quite a while now. I recently tried the evaluate_detections() method with fiftyone.core.labels.Detections and COCO evaluation method, which is very powerful.

When I upgraded Fiftyone to 1.1.0 (and also in 1.2.0), I've noticed a strange behavior that I can not explain.

It looks like some predictions are not evaluated. Their eval_id remains empty, and the detection is neither evaluated as a fp or tp. When investigated, I discovered it seems it was all the predictions that are False Positive and where there is also a GT on the same image ( a non-negative image).

Code to reproduce issue

This is how I created the evaluation :
dataset = fo.load_dataset('my_dataset')

dataset.evaluate_detections(
            pred_field='preds',
            gt_field="ground_truth",
            eval_key=f"eval_test",
            classwise=False,
            compute_mAP=False,
            iou=0.5,
        )

System information

• OS Platform and Distribution : Linux Ubuntu 22.04
• Python version (python --version): Python 3.10.13
• FiftyOne version (fiftyone --version): 1.2.0
• FiftyOne installed from (pip or source): poetry

Other info/logs

Include any logs or source code that would be helpful to diagnose the problem.
If including tracebacks, please include the full traceback. Large logs and
files should be attached. Please do not use screenshots for sharing text. Code
snippets should be used instead when providing tracebacks, logs, etc.

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another
member of your organization be willing to contribute a fix for this bug to the
FiftyOne codebase?

• Yes. I would be willing to contribute a fix for this bug with guidance
  from the FiftyOne community
• No. I cannot contribute a bug fix at this time

[mwoodson1]

Can you provide any more details on how to reproduce the issue, what kind of data, labels, etc? I tried reproducing this using coco-2017 validation set and faster-rcnn predictions. With that data and predictions I can see many False Positives but no samples ended up with empty eval_ids.

[AntoninDuvalPreligens]

Hi @mwoodson1. Unfortunatly, I can't share directly the data. I can share however a sample which had this issue.

I ran two evaluations, one with fiftyone 1.0.2 installed, using key eval_model . I re-ran the evaluation with fiftyone==1.2.0 and key eval_model_bug.

Here is one of the sample where eval_model_bug is missing for some detections.

<Sample: {
    'id': '6776a8c70f0ac1cade9f4153',
    'media_type': 'image',
    'filepath': 'path',
    'tags': ['label_1'],
    'metadata': None,
    'ground_truth': <Detections: {
        'detections': [
            <Detection: {
                'id': '6776a8ac0f0ac1cade9eff0b',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.921875, 0.248046875, 0.078125, 0.125],
                'mask': None,
                'mask_path': None,
                'confidence': None,
                'index': None,
                'eval_model': 'fn',
                'eval_model_id': '',
                'eval_model_bug': 'fn',
                'eval_model_bug_id': '',
            }>,
            <Detection: {
                'id': '6776a8ac0f0ac1cade9eff0c',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.8564453125, 0.818359375, 0.1435546875, 0.1328125],
                'mask': None,
                'mask_path': None,
                'confidence': None,
                'index': None,
                'eval_model': 'tp',
                'eval_model_id': '6776aa66aa9d2931e6823bee',
                'eval_model_iou': 0.8132716049382716,
                'eval_model_bug': 'tp',
                'eval_model_bug_id': '6776aa66aa9d2931e6823bee',
                'eval_model_bug_iou': 0.8132716049382716,
            }>,
        ],
    }>,
    'model': <Detections: {
        'detections': [
            <Detection: {
                'id': '6776aa66aa9d2931e6823bee',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.8623046875, 0.8154296875, 0.12109375, 0.138671875],
                'mask': None,
                'mask_path': None,
                'confidence': 0.9986949563026428,
                'index': None,
                'prediction_name': 'label_1',
                'eval_model': 'tp',
                'eval_model_id': '6776a8ac0f0ac1cade9eff0c',
                'eval_model_iou': 0.8132716049382716,
                'eval_model_bug': 'tp',
                'eval_model_bug_id': '6776a8ac0f0ac1cade9eff0c',
                'eval_model_bug_iou': 0.8132716049382716,
            }>,
            <Detection: {
                'id': '6776aa66aa9d2931e6823bef',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.0908203125, 0.455078125, 0.14453125, 0.1171875],
                'mask': None,
                'mask_path': None,
                'confidence': 0.633573591709137,
                'index': None,
                'prediction_name': 'label_1',
                'eval_model': 'fp',
                'eval_model_id': '',
                'eval_model_bug_id': '',
            }>,
            <Detection: {
                'id': '6776aa66aa9d2931e6823bf0',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.0546875, 0.216796875, 0.1552734375, 0.115234375],
                'mask': None,
                'mask_path': None,
                'confidence': 0.44886577129364014,
                'index': None,
                'prediction_name': 'label_1',
                'eval_model': 'fp',
                'eval_model_id': '',
                'eval_model_bug_id': '',
            }>,
            <Detection: {
                'id': '6776aa66aa9d2931e6823bf1',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.0224609375, 0.3681640625, 0.1591796875, 0.1181640625],
                'mask': None,
                'mask_path': None,
                'confidence': 0.3553047776222229,
                'index': None,
                'prediction_name': 'label_1',
                'eval_model': 'fp',
                'eval_model_id': '',
                'eval_model_bug_id': '',
            }>,
            <Detection: {
                'id': '6776aa66aa9d2931e6823bf2',
                'attributes': {},
                'tags': [],
                'label': 'label_1',
                'bounding_box': [0.046875, 0.607421875, 0.1669921875, 0.123046875],
                'mask': None,
                'mask_path': None,
                'confidence': 0.9848335385322571,
                'index': None,
                'prediction_name': 'label_1',
                'eval_model': 'fp',
                'eval_model_id': '',
                'eval_model_bug_id': '',
            }>,
        ],
    }>,
    'eval_model_tp': 1,
    'eval_model_fp': 4,
    'eval_model_fn': 1,
    'eval_model_bug_tp': 1,
    'eval_model_bug_fp': 0,
    'eval_model_bug_fn': 1,
}>