How to save checkpoints within lightning_logs?

[polars05]

I'm currently doing checkpointing as follows:

checkpoint_callback = pl.callbacks.ModelCheckpoint(
          filepath=os.path.join(os.getcwd(), 'checkpoints/{epoch}-{val_loss:.2f}'),
          verbose=True,
          monitor='val_loss', 
          mode='min', 
          save_top_k=-1,
          period=1
      )

 
  trainer = pl.Trainer(
      default_save_path=os.path.join(os.getcwd(), 'log_files_are_stored_here'),
      gpus=1,
      max_epochs=2
      checkpoint_callback=checkpoint_callback
  )  

This creates the following folder structure:

├── checkpoints # all the .pth files are saved here
└── log_files_are_stored_here
    └── lightning_logs 
       ├── version_0
       ├── version_1
       ├── version_2

How can I get the .pth files for each version to be saved in the respective version folders like so?:

└── log_files_are_stored_here
    └── lightning_logs 
       ├── version_0
            └── checkpoints #  save the .pth files here
       ├── version_1
            └── checkpoints #  save the .pth files here
       ├── version_2
            └── checkpoints #  save the .pth files here

[chris-clem]

Hi,

this is how I do it:

tt_logger = TestTubeLogger(save_dir=str(log_dir / "tt_logs"), name=run_name)

checkpoint_dir = (
    Path(tt_logger.save_dir)
    / tt_logger.experiment.name
    / f"version_{tt_logger.experiment.version}"
    / "checkpoints"
)
filepath = checkpoint_dir / "{epoch}-{val_loss:.4f}"
checkpoint_cb = ModelCheckpoint(filepath=str(filepath))

trainer = pl.Trainer(
        logger=tt_logger,
        checkpoint_callback=checkpoint_cb,
        ...
    )

[oplatek]

Hi, the TensorBoard version inspired by @chris-clem snippet.

Any idea how to get rid of the "HACK"?

tb_logger = TensorBoardLogger(save_dir='logs/tb/')

# HACK: to avoid tb_logger crashing in self._get_next_version() if I access tb_logger.log_dir
os.makedirs(f'logs/tb/default', exist_ok=True) 

mcp =  ModelCheckpoint(filepath=f'{tb_logger.log_dir}/' + '{epoch}_vl_{val_loss:.2f}')
trainer = Trainer(logger=tb_logger, checkpoint_callback=mcp) 
...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vedal]

@Borda Could we re-open this?
The current defaults of

ModelCheckpoint.dirpath = ${trainer.default_root_dir}/checkpoints   
TensorBoardLogger.save_dir = ${trainer.default_root_dir}/lightning_logs/version_i/

seem to me a bit unintuitive, because it decouples the checkpoints from stored configs and logs.

It would be really handy to have the structure @polars05 proposed as default. Maybe one option could be a new argument experiment_name to trainer and in trainer.__init__ run
self.version_id = self._get_next_version()

such that new defaults are:

ModelCheckpoint.dirpath = ${trainer.default_root_dir}/${trainer.experiment_name}/${trainer.version_id}/checkpoints   
TensorBoardLogger.save_dir = ${trainer.default_root_dir}/${trainer.experiment_name}/${trainer.version_id}/logs

or similar ?

[vedal]

@Borda Thank you

It seems @awaelchli is working to standardize a self.experiment_dir across logger base class! #14188

Another auto-closed issue which seems to access the exact log dir (including version_id):
#2409

[vedal]

My bad, there seems to already be a way to do this:
if using ModelCheckpoint and TensorboardLogger, then setting the TensorboardLogger.save_dir does what @polars05 suggested.

For reference, this is how to set this in a .yaml for LightningCLI

directory: my/output/dir
trainer:
    #default_root_dir: 
    callbacks:
    - class_path: ModelCheckpoint # short-hand notation
      init_args:
        #dirpath: checkpoints
    logger:
        class_path: pytorch_lightning.loggers.TensorBoardLogger
        init_args:
            save_dir: ${directory}
            name: some_experiment_name

That is:

1. leave Trainer.default_root_dir as default
2. leave ModelCheckpoint.dir_path as default
3. set TensorBoardLogger.save_dir to the desired output path

I suggest that this should be clearly specified in the documentation for Trainer.default_root_dir.

@Borda We can close this issue now, and I'll report it in #14188. Sorry for the disturbance