Question about the intended behavior of Accuracy with average="none" and nan results

[phiyodr]

Discussed in #466

^{Originally posted by phiyodr August 19, 2021}
Hi,

I'm using your Accuracy metric without averaging and have difficulties with nan results, as I want to process the tensor further.

Example with 4 binary labels (cat, dog, horse, cow) and 3 samples:

from torchmetrics import Accuracy # version '0.5.0'
accs = Accuracy(average="none", num_classes=4)

probs  = torch.tensor([[.9, 0.1, 0.9, 0.1], 
                       [.8, 0.2, 0.9, 0.2], 
                       [.7, 0.1, 0.8 , 0.2]])

targets = torch.tensor([[1, 1, 0, 0], 
                        [1, 1, 0, 0], 
                        [1, 1, 0, 0]])

accs(probs, targets)
#> tensor([1., 0., 0., nan])

On your Docs you write that this is intended: "If 'none' and a given class doesn’t occur in the preds or target, the value for the class will be nan."

But technically the last class has an Accuracy of 1.. Statistics like min or max do not work on tensors with nan. My real model has some thousand labels and imputing nan with 1. doesn't feel that good ;)

Internally you treat the last label with an Accuracy of 1.:

acc = Accuracy(num_classes=4)
acc(probs, targets)
#> tensor(0.5000)

To some extend I understand your thoughts, but I want to ask if this is really the intended result or if an argument like suppress_nan is possible.
Thanks for your module!

Best
Philipp