Handling AttributeErrors while cleaning params namespace while setting up fit

[chiragraman]

🐛 Bug

is_picklable in parsing.py does not handle AttributeError thrown by pickle.dumps() - specifically, the following :
AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

To Reproduce

Here's a stack trace:

Traceback (most recent call last):
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/chirag/Projects/mingle/social-processes/run/run_synthetic_social.py", line 120, in <module>
    main()
  File "/home/chirag/Projects/mingle/social-processes/run/run_synthetic_social.py", line 116, in main
    trainer.fit(process, datamodule=dm)
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 425, in fit
    self.train_loop.setup_fit(model, train_dataloader, val_dataloaders, datamodule)
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 90, in setup_fit
    parsing.clean_namespace(model.hparams)
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/parsing.py", line 75, in clean_namespace
    del_attrs = [k for k, v in hparams_dict.items() if not is_picklable(v)]
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/parsing.py", line 75, in <listcomp>
    del_attrs = [k for k, v in hparams_dict.items() if not is_picklable(v)]
  File "/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/parsing.py", line 62, in is_picklable
    pickle.dumps(obj)
AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

I forked the repo, found the file, and made the following change to is_picklable:

def is_picklable(obj: object) -> bool:
    """Tests if an object can be pickled"""

    try:
        pickle.dumps(obj)
        return True
    except (pickle.PicklingError, AttributeError):
        return False

I then installed the package from my local repo, ran the same code and got the following warnings:

/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: attribute 'trials' removed from hparams because it cannot be pickled
  warnings.warn(*args, **kwargs)
/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: attribute 'optimize_parallel' removed from hparams because it cannot be pickled
  warnings.warn(*args, **kwargs)
/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: attribute 'optimize_parallel_gpu' removed from hparams because it cannot be pickled
  warnings.warn(*args, **kwargs)
/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: attribute 'optimize_parallel_cpu' removed from hparams because it cannot be pickled
  warnings.warn(*args, **kwargs)
/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: attribute 'generate_trials' removed from hparams because it cannot be pickled
  warnings.warn(*args, **kwargs)
/home/chirag/miniconda3/envs/ml/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: attribute 'optimize_trials_parallel_gpu' removed from hparams because it cannot be pickled
  warnings.warn(*args, **kwargs)

These aren't params added by the end user I believe, and for some reason pickle raises an AttributeError rather than pickling.PicklingError for these.

Environment

• CUDA:
  - GPU:
  - Quadro P4000
  - available: True
  - version: 10.2
• Packages:
  - numpy: 1.19.1
  - pyTorch_debug: False
  - pyTorch_version: 1.6.0
  - pytorch-lightning: 0.9.1rc4
  - tqdm: 4.48.2
• System:
  - OS: Linux
  - architecture:
  - 64bit
  - ELF
  - processor: x86_64
  - python: 3.8.5
  - version: Support of different batch types #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020

Additional context

Not sure if the solution is to also handle AttributeError, or if a more elegant alternative is needed.

[chiragraman]

I believe these are added when using the HyperOptArgumentParser from test_tube.

[edenlightning]

Thanks for the issue @chiragraman! Any way you can send a PR to fix?

[chiragraman]

@edenlightning sure thing! Is the suggested solution to also catch AttributeErrors alright?

[edenlightning]

@Borda might have opinions on this