Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensorboard logger fails to save model OmegaConf hparams #2844

Closed
ananthsub opened this issue Aug 6, 2020 · 0 comments · Fixed by #2846 or #2519
Closed

Tensorboard logger fails to save model OmegaConf hparams #2844

ananthsub opened this issue Aug 6, 2020 · 0 comments · Fixed by #2846 or #2519
Assignees
Labels
bug Something isn't working help wanted Open to be worked on

Comments

@ananthsub
Copy link
Contributor

🐛 Bug

The Tensorboard logger fails to log module hyperparameters configured with OmegaConf. This happens when updating the logger hparams here:

  • The trainer calls the logger's log_hyperparams here:
  • Inside log_hyperparams the logger's hparams are updated here. This causes the hparams type to now be dict instead of DictConfig
  • As a result, this branch in [save_hparams_to_yaml](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/core/saving.py#L330-L333) is never triggered

This is the stacktrace when logging hyperparams: https://gist.github.com/ananthsub/7acfdb0e0f551ed030f05f7674c37b46

To Reproduce

Code sample

A hacky fix would be something like changing the hparams update to use this inside the tensorboard logger:

if isinstance(params, Container):
   self.hparams = OmegaConf.merge(self.hparams, params)
else:
    self.hparams.update(params)

Expected behavior

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py
  • PyTorch Version (e.g., 1.0):
  • OS (e.g., Linux):
  • How you installed PyTorch (conda, pip, source):
  • Build command you used (if compiling from source):
  • Python version:
  • CUDA/cuDNN version:
  • GPU models and configuration:
  • Any other relevant information:

Additional context

@ananthsub ananthsub added bug Something isn't working help wanted Open to be worked on labels Aug 6, 2020
@ananthsub ananthsub changed the title Tensorboard logger fails to save model Omegaconf hparams Tensorboard logger fails to save model OmegaConf hparams Aug 6, 2020
williamFalcon pushed a commit that referenced this issue Aug 7, 2020
* Add support to Tensorboard logger for OmegaConf hparams

Address #2844

We check if we can import omegaconf, and if the hparams are omegaconf instances. if so, we use OmegaConf.merge to preserve the typing, such that saving hparams to yaml actually triggers the OmegaConf branch

* avalaible

* chlog

* test

Co-authored-by: Jirka Borovec <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants