logging interval seems not working when i check results in Tensorboard

[QW-Is-Here]

Discussed in #11460

^{Originally posted by MrWangg1992 January 13, 2022}
Hi ,
i set all the settings with log interval == 'epoch' as the code shown below, but when i check the results output in tensorboard, it seems like still logging in steps, may i kindly ask how to figure it out?

lr_monitor = LearningRateMonitor(logging_interval='epoch')



    def configure_optimizers(self):
      
            optimizer_init = getattr(torch.optim, self.opt)
            optimizer = optimizer_init(self.parameters(), lr=self.basic_lr)


            scheduler_init = getattr(torch.optim.lr_scheduler, self.scheduler)
            lr_scheduler = scheduler_init(optimizer=optimizer)

        lr_scheduler_config = {
            # REQUIRED: The scheduler instance
            "scheduler": lr_scheduler,
            # The unit of the scheduler's step size, could also be 'step'.
            # 'epoch' updates the scheduler on epoch end whereas 'step'
            # updates it after a optimizer update.
            "interval": "epoch",
            # How many epochs/steps should pass between calls to
            # `scheduler.step()`. 1 corresponds to updating the learning
            # rate after every epoch/step.
            "frequency": 1,
            # Metric to monitor for schedulers like `ReduceLROnPlateau`
            "monitor": "val_loss",
            # If set to `True`, will enforce that the value specified 'monitor'
            # is available when the scheduler is updated, thus stopping
            # training if not found. If set to `False`, it will only produce a warning
            "strict": True,
            # If using the `LearningRateMonitor` callback to monitor the
            # learning rate progress, this keyword can be used to specify
            # a custom logged name
            "name": 'LearningRateMonitor',
        }

        return {"optimizer": optimizer, "lr_scheduler": lr_scheduler_config}


    def validation_step(self, val_batch, batch_idx):
        self.head.train()
        x, y, _, _ = val_batch
        fpn_outs = self.backbone(x)
        loss, _, _, _, _, _ = self.head(fpn_outs, y, x)
        self.log('loss/val', loss,  prog_bar=True, on_step=False, on_epoch=True)

    def training_epoch_end(self, outputs):
        avg_loss = torch.stack([x['loss'] for x in outputs]).mean()
        self.log('train-loss/epoch', avg_loss, prog_bar=True, on_step=False, on_epoch=True)


    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['loss'] for x in outputs]).mean()
        self.log('valid-loss/epoch', avg_loss, prog_bar=True, on_step=False, on_epoch=True)
```</div>

cc @rohitgr7

[rohitgr7]

hey @MrWangg1992

the x-axis is always the global_step. When you hover the graph of lr_monitor, does the pointer move at every step?

[QW-Is-Here]

hover

hi @rohitgr7

I check the graph: 1 epoch nearly equals 1k steps

The cursor jumps every 1k steps, so that means all the settings here are correct when I want to do the training based on epochs? It's kind of confusing, i wish to update my lr by epoch, but it looks like updated by steps when it shown in the Tensor-board.

And would you mind teach me how to change the x-axis format from steps to epochs?

[rohitgr7]

hey @MrWangg1992!

It's kind of confusing, i wish to update my lr by epoch, but it looks like updated by steps when it shown in the Tensor-board.

well yes, this is globally set that all the logs will be logged with global_step as the x-axis by default for consistency.

And would you mind teach me how to change the x-axis format from steps to epochs?

I think it's not possible with tensorboard with the default implementation because it doesn't allow configuring the graphs once logged. You can try some sort of advanced logger though, like WandbLogger, that gives you the option to update x-axis once it's logged in their UI.

If you want to stick to tensorboard, then you override LearningRateMonitor to make it work as per your requirement.

from pytorch_lightning.callbacks import LearningRateLogger

class CustomLearningRateMonitor(LearningRateMonitor):
    def on_train_epoch_start(self, trainer: "pl.Trainer", *args: Any, **kwargs: Any) -> None:
        assert trainer.logger is not None
        if self.logging_interval != "step":
            interval = "epoch" if self.logging_interval is None else "any"
            latest_stat = self._extract_stats(trainer, interval)

            if latest_stat:
                trainer.logger.log_metrics(latest_stat, step=trainer.current_step)  # <-- change global_step -> current_epoch

trainer = Trainer(..., callbacks=[..., CustomLearningRateMonitor(...)])

[QW-Is-Here]

hey @MrWangg1992!

It's kind of confusing, i wish to update my lr by epoch, but it looks like updated by steps when it shown in the Tensor-board.

well yes, this is globally set that all the logs will be logged with global_step as the x-axis by default for consistency.

And would you mind teach me how to change the x-axis format from steps to epochs?

I think it's not possible with tensorboard with the default implementation because it doesn't allow configuring the graphs once logged. You can try some sort of advanced logger though, like WandbLogger, that gives you the option to update x-axis once it's logged in their UI.

If you want to stick to tensorboard, then you override LearningRateMonitor to make it work as per your requirement.

from pytorch_lightning.callbacks import LearningRateLogger

class CustomLearningRateMonitor(LearningRateMonitor):
    def on_train_epoch_start(self, trainer: "pl.Trainer", *args: Any, **kwargs: Any) -> None:
        assert trainer.logger is not None
        if self.logging_interval != "step":
            interval = "epoch" if self.logging_interval is None else "any"
            latest_stat = self._extract_stats(trainer, interval)

            if latest_stat:
                trainer.logger.log_metrics(latest_stat, step=trainer.current_step)  # <-- change global_step -> current_epoch

trainer = Trainer(..., callbacks=[..., CustomLearningRateMonitor(...)])

@rohitgr7
Thank you so much for quick responding! I wish to update my learning rate base on epoch, am I doing it in a right way by use the setting codes above? Also I will try rewrite the monitor and see if it works!

[rohitgr7]

yeah! rest is correct.

[rohitgr7]

closing this issue. Feel free to reopen :)