Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

torchmetrics.F1 bug with ignore_index and macro averaging #486

Closed
ashispati opened this issue Aug 29, 2021 · 2 comments · Fixed by #495
Closed

torchmetrics.F1 bug with ignore_index and macro averaging #486

ashispati opened this issue Aug 29, 2021 · 2 comments · Fixed by #495
Assignees
Labels
bug / fix Something isn't working help wanted Extra attention is needed

Comments

@ashispati
Copy link

🐛 Bug

An error is raised while using torchmetrics.F1 if:

  1. ignore_index set to a valid class index (< num_classes)
  2. `average = "macro"
  3. one of the non-ignored classes has no positive values in the target array

To Reproduce

Run the following code:

from torchmetrics import F1
import torch 

target = torch.tensor([0, 0, 0, 0])
preds = torch.tensor([0, 0, 0, 0])

# case 1: not using `ignore_index`
f1 = F1(num_classes=3, average="macro")
f1(preds, target)  # output:  tensor(1.)

# case 2: using `ignore_index`
f1 = F1(num_classes=3, average="macro", ignore_index=2)
f1(preds, target)  # raises Error

The following error in raised in torchmetrics/functional/classification/f_beta.py:
line 100, in _fbeta_compute num[ignore_index, ...] = -1 IndexError: index 2 is out of bounds for dimension 0 with size 2

Expected behavior

The result in the above example should be the same as the case with no ignore_index provided.

Environment

  • PyTorch Version (e.g., 1.0): 1.8.1
  • Python version: 3.6.12
  • CUDA/cuDNN version: NA
  • GPU models and configuration: NA
  • torchmetrics version: 0.5.0
@ashispati ashispati added bug / fix Something isn't working help wanted Extra attention is needed labels Aug 29, 2021
@github-actions
Copy link

Hi! thanks for your contribution!, great first issue!

@ashispati ashispati changed the title torchmetrics.F1 bug with ingore_index and macro averaging torchmetrics.F1 bug with ignore_index and macro averaging Aug 29, 2021
@SkafteNicki
Copy link
Member

Hi @ashispati,
I created PR #495 that should fix the issue.

@Borda Borda closed this as completed in #495 Sep 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants