AUROC for Multi-Label Classification with binary targets

[jfltzngr]

Hi everyone,
I can't figure out how to perform AUROC on my Multi-Target/Multi-Label Classification problem. My predictions are probabilities that do not sum up to one (Multi-Target). The Target is One-Hot/binary encoded.
For example with N=3 (batch_size) and C=5(num_classes):

predictions:

tensor([[0.5456, 0.5710, 0.3132, 0.1019, 0.6820],
[0.2719, 0.1826, 0.7519, 0.8385, 0.9649],
[0.7703, 0.7506, 0.7581, 0.2897, 0.4976]], device='cuda:0',
dtype=torch.float64)

target:

tensor([[1, 0, 1, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 1, 1, 1]], device='cuda:0')

I initialize the AUROC:

metric_auroc = torchmetrics.AUROC(num_classes=5)
metric_auroc.to(device)
auroc = metric_aucroc(preds, target)

I get the following ValueError:
ValueError: No positive samples in targets, true positive value should be meaningless

Is there a way to implement the exisitng AUROC on my problem?

Thanks in advance to your commitment!

[SkafteNicki]

in your second column of your target tensor you have pure zeros, which means that for that label we cannot compute the auroc score because the score is undefined when we do not have both 0 and 1 in the target tensor

[jfltzngr]

Thanks a lot!

[monk1337]

I am also facing the same issue. I am using only those samples where at least one example is positive, and my classes are pretty large ( ~250 ). What is the possible solution for this?

[SkafteNicki]

@monk1337 do you batch contain both 0 and 1? because then you should not get this error

[monk1337]

Hi, Yes, I have removed the samples which contain only 0. I didn't get the point when you said, "column of your target tensor you have pure zeros" it should calculate the AUC row-wise per sample, right?

[jfltzngr]

Hi @monk1337, in my case the rows represent the batches and the columns the classes.
You can calculate the auroc class-wise/column-wise -> for each class, and for the whole batch averaged. Depending on your way of averaging, the auc will be calculated under one roc reagarding all TP and FP of the whole batch, or it calculates the auroc for each class/column-wise and averages it by the number of classes afterwards. This is how I understood the AUROC, correct me if I am wrong.

[SkafteNicki]

@monk1337 it is always assumed that the row dimension is the batch size and the column dimension in the label dimension. Thus a single column would correspond to a batch for a single label. Therefore, each column needs to have both 0 and 1.

However, if you evaluate AUROC only on epoch end you should not run into these kind of problems.

[monk1337]

@SkafteNicki @jfltzngr I get your point, Here I am explaining my problem with example

So if my train data looks like this:

I only keep those rows where at least one column is 1

Now as you said there should be no all zeros in any column too, so I filtered the samples by column-wise too and removed those where all rows are 0 of a particular label

This is filtered train data with three labels left, but my test data looks like this with old 4 labels :

Now I can't train the model on this because train data size Is batch x3 while test data size is batch x 4 How you'll deal this kind of problem for AUROC?

In real dataset if I remove all zero columns then my train dataset is 47000 x 8504 , test dataset is 12000 x 4304 and val dataset is 10000 x 2300