DDP Accumulate predictions from all processes across all nodes to compute accuracy

[Gateway2745]

Hi. I'm using pytorch-lightning and would like to calculate the classification accuracy (#right / #total) where #right = correct predictions from all processes, and #predictions = total no. of predictions when using DDP. I do NOT want to compute accuracies from individual processes and then average them. I want to get the total right predictions and then finally compute the accuracy. Currently I am doing this.

class Model(pl.LightningModule):
    def __init__(self):
        super(Model, self).__init__()
        self.accuracy = torchmetrics.Accuracy()

    def validation_step(self, batch, batch_idx):
        y = ...
        y_pred = ...
        self.accuracy(y_pred, y)

    def validation_epoch_end(self, validation_step_outputs):
        self.log('val_acc_epoch', self.accuracy.compute())

However, when I compare the output of self.accuracy.compute() to manual computation (by outputting right and total no. preds from the individual processes) they are not the same. What is the correct way to get my expected output?

As, I understand it self.accuracy(y_pred, y) accumulates the global state but returns the local metrics. self.accuracy.compute() returns the metrics on the global state after syncing and reducing. But I notice in my logs that self.accuracy.compute() continuously increases indicating that the global metrics are not being reset. But the docs mention that they are reset after every epoch.

Thank you!

[SkafteNicki]

Hi,
The docs state that the metric is reset IF the metric is directly logged using self.log, something like this:

self.log('acc', self.accuracy)  # log the object not the return value

however, the easiest thing for you to do is to manually just reset the metric by adding self.accuracy.reset() as the last statement to your validation_epoch_end method.

[Gateway2745]

Ohh I see, thank you. It works now!