Measure a metric over the entire test dataset

[celsofranssa]

Hello,

I normally apply TorchMetrics combined with PL as following:

class PLModel(pl.LightningModule):

    def __init__(self, hparams):
        super(PLModel, self).__init__()
        self.save_hyperparameters(hparams)

        self.encoder = instantiate(hparams.encoder)

        self.cls_head = torch.nn.Sequential(
            torch.nn.Dropout(hparams.dropout),
            torch.nn.Linear(hparams.hidden_size, hparams.num_classes),
            torch.nn.LogSoftmax(dim=-1)
        )

        # metrics
        self.val_metrics = self._get_metrics(prefix="test_")
        ...

    def _get_metrics(self, prefix):
        return MetricCollection(
            metrics={
                "Mic-F1": F1(num_classes=self.hparams.num_classes, average="micro"),
                "Wei-F1": F1(num_classes=self.hparams.num_classes, average="weighted")
            },
            prefix=prefix)
    ...

    def test_step(self, batch, batch_idx):
        text, true_cls = batch["text"], batch["cls"]
        pred_cls = self.cls_head(
            self(text)
        )

        # log text metrics
        self.log_dict(self.test_metrics(pred_cls, true_cls), prog_bar=True)

In this scenario, what is the proper approach to calculate the F1 metric over the entire test dataloader instead of per batch?

[celsofranssa]

It is automatic done when the test_epoch_end is implemented in PL?

    def test_epoch_end(self, outs):
        self.test_metrics.compute()

[SkafteNicki]

I would recommend something like this:

def test_step(self, batch, batch_idx):
    text, true_cls = batch["text"], batch["cls"]
    pred_cls = self.cls_head(
        self(text)
    )
    # just update states, no logging
    self.test_metrics.update(pred_cls, true_cls))


def test_epoch_end(self, outs):
    self.log_dict(self.test_metrics.compute())

then it is very clear that you update the metric on all test data and only compute at the very end.

[wellhowtosay]

I would recommend something like this:

def test_step(self, batch, batch_idx):
    text, true_cls = batch["text"], batch["cls"]
    pred_cls = self.cls_head(
        self(text)
    )
    # just update states, no logging
    self.test_metrics.update(pred_cls, true_cls))


def test_epoch_end(self, outs):
    self.log_dict(self.test_metrics.compute())

then it is very clear that you update the metric on all test data and only compute at the very end.

Does this work as expected when DDP and multi gpus were selected? Because I need the metric on whole dev/test set

[SkafteNicki]

@wellhowtosay yes it will also work with DDP+multi gpus.
When .compute() is called we sync the metric states between the different device, and compute the metric value on the combine state from all processes.

[wellhowtosay]

@wellhowtosay yes it will also work with DDP+multi gpus.
When .compute() is called we sync the metric states between the different device, and compute the metric value on the combine state from all processes.

Thx bro !