MeanAveragePrecision doesn't work as expected when using max_detection_thresholds != [1, 10, 100]

[r-remus]

🐛 Bug

When using MeanAveragePrecision, the mARs aren't computed as expected when max_detection_thresholds are not the "default" values of [1, 10, 100]. E.g., when using [1, 10, 1000] (using the pycocotools backend) or when using [1, 10, 100, 1000] (using the faster_coco_eval backend), still only mAR@1, mAR@10, and mAR@100 are computed.

To Reproduce

Use this code snippet to reproduce the bug:

import torch
from torchmetrics.detection import MeanAveragePrecision

map_pycocotools = MeanAveragePrecision(
    box_format='xyxy',
    iou_type='bbox',
    class_metrics=True,
    max_detection_thresholds=[1, 10, 1000],
    backend='pycocotools',
)

map_faster_coco_eval = MeanAveragePrecision(
    box_format='xyxy',
    iou_type='bbox',
    class_metrics=True,
    max_detection_thresholds=[1, 10, 100, 1000],
    backend='faster_coco_eval',
)

predictions = [{
    'boxes': torch.Tensor([[1.01, 2.02, 3.03, 4.04], [5.05, 6.06, 7.07, 8.08]]),
    'labels': torch.Tensor([0, 1]).to(torch.int32),
    'scores': torch.Tensor([1.0, 1.0]),
}]
targets = [{
    'boxes': torch.Tensor([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]),
    'labels': torch.Tensor([0, 1]).to(torch.int32),
}]

map_pycocotools.update(predictions, targets)
map_faster_coco_eval.update(predictions, targets)

results_pycocotools = map_pycocotools.compute()
results_faster_coco_eval = map_faster_coco_eval.compute()
print(results_pycocotools)
print(results_faster_coco_eval)

Running the code above results in the following print out

{'map': tensor(-1.), 'map_50': tensor(1.), 'map_75': tensor(1.), 'map_small': tensor(0.9000), 'map_medium': tensor(-1.), 'map_large': tensor(-1.), 'mar_1': tensor(0.9000), 'mar_10': tensor(0.9000), 'mar_100': tensor(0.9000), 'mar_small': tensor(0.9000), 'mar_medium': tensor(-1.), 'mar_large': tensor(-1.), 'map_per_class': tensor([-1., -1.]), 'mar_100_per_class': tensor([1.0000, 0.8000]), 'classes': tensor([0, 1], dtype=torch.int32)}
{'map': tensor(0.9000), 'map_50': tensor(1.), 'map_75': tensor(1.), 'map_small': tensor(0.9000), 'map_medium': tensor(-1.), 'map_large': tensor(-1.), 'mar_1': tensor(0.9000), 'mar_10': tensor(0.9000), 'mar_100': tensor(0.9000), 'mar_small': tensor(0.9000), 'mar_medium': tensor(-1.), 'mar_large': tensor(-1.), 'map_per_class': tensor([1.0000, 0.8000]), 'mar_100_per_class': tensor([1.0000, 0.8000]), 'classes': tensor([0, 1], dtype=torch.int32)}

where we can clearly see that mAR@1000 has not been compuzted, irrespective of the used backend.

Expected behavior

I expect that I can compute mARs for the the given max_detection_thresholds, no matter what these thresholds actually are.

Environment

• TorchMetrics version 1.3.0 (installed via pip)
• Python & PyTorch Version (e.g., 1.0): Python 3.10.12, PyTorch 2.2.0
• Any other relevant information such as OS (e.g., Linux): Ubuntu 22.04.3 LTS

Additional context

This was working fine for the pycocotools backend in torchmetrics 0.11.4. This even worked for max_detection_thresholds = [1, 10, 100, 10000], which apparently isn't possible anymore, because only len(max_detection_thresholds) == 3 is allowed when using the pycocotools backend.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

@r-remus thanks for reporting this issue.
The issue will be fixed in PR #2367. Some good news and bad news about the fix:

• When len(max_detection_thresholds) == 3 it should be fixed such that the returned statistics correctly corresponds to whatever custom threshold you provide
• I realized that for len(max_detection_thresholds)!=3 that neither backends actually support this. There is really nothing we can do about it when the different frameworks does not support it. Therefore moving forward max_detection_thresholds will need have length 3.

[r-remus]

Thanks, that's great!