UserWarning: Metric AUROC will save all targets and predictions in buffer.

[ayoubassis]

Hi,

I'm training a model on a large dataset, and assessing its performance using AUROC. This causes the training to break after a few epochs due to out of memory issue related to this warning:

UserWarning: Metric AUROC will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.

I'm wondering if it's possible not to store all prediction and ground truth values, and get rid of them after computing the metric ( especially when compute_on_step is set to true).

[SkafteNicki]

Hi @blurry-mood, are you resetting the metric between epochs:

def training/validation_epoch_end(self):
    val = self.auroc_metric.compute()
    self.log("train/val_auroc", val)
    self.auroc_metric.reset()  # reset before next epoch

if not you are accumulating over all epochs which will lead to OOM.

[ayoubassis]

if not you are accumulating over all epochs which will lead to OOM.

Yeah, this was the case.

Thank you for denoting the .reset() method.
It would've been nice is this was done automatically by torchmetrics, by passing an argument to the constructor.

[SkafteNicki]

No probs :]
We have a open issue requesting this: #482.
Will think about adding this to torchmetrics.