Infinite recursion when calling self.log(...) in validation loop with dataset that returns string in item dict

[nathanpainchaud]

🐛 Bug

I'm not sure if this is a behavior that was intended to be supported in the first place, but PR #3888 introduced a regression on passing strings as part of the data in a batch. Now, if we pass a dictionary where one of the values is a string, Result.unpack_batch_size falls into an infinite recursion loop when trying to log anything during the validation step.

To Reproduce

See PR #3907 for a test that reproduces the bug on the current master. The test becomes functional when commenting out line 22 (the call to self.log(...) in the validation loop).

The recursion happens in this specific statement in unpack_batch_size():

elif isinstance(sample, Iterable):
    sample = next(iter(sample), 1)
    size = self.unpack_batch_size(sample)

which recurses infinitely when sample is a string.

The full stacktrace I get when running the test is the following:

test_logging.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../pytorch_lightning/trainer/trainer.py:487: in fit
    results = self.accelerator_backend.train()
../../pytorch_lightning/accelerators/cpu_backend.py:47: in train
    results = self.train_or_test()
../../pytorch_lightning/accelerators/base_backend.py:47: in train_or_test
    results = self.trainer.train()
../../pytorch_lightning/trainer/trainer.py:509: in train
    self.run_sanity_check(self.get_model())
../../pytorch_lightning/trainer/trainer.py:699: in run_sanity_check
    _, eval_results = self.run_evaluation(test_mode=False, max_batches=self.num_sanity_val_batches)
../../pytorch_lightning/trainer/trainer.py:632: in run_evaluation
    step_metrics = self.evaluation_loop.log_evaluation_step_metrics(batch, batch_idx)
../../pytorch_lightning/trainer/evaluation_loop.py:323: in log_evaluation_step_metrics
    results.track_batch_size(batch)
../../pytorch_lightning/core/step_result.py:222: in track_batch_size
    batch_size = self.unpack_batch_size(batch)
../../pytorch_lightning/core/step_result.py:335: in unpack_batch_size
    size = self.unpack_batch_size(sample)
../../pytorch_lightning/core/step_result.py:338: in unpack_batch_size
    size = self.unpack_batch_size(sample)
../../pytorch_lightning/core/step_result.py:338: in unpack_batch_size
    size = self.unpack_batch_size(sample)
E   RecursionError: maximum recursion depth exceeded in comparison
!!! Recursion detected (same locals & position)

Code sample

See See PR #3907.

Expected behavior

We should be able to use strings in the data returned by the dataset, and still be able to call self.log(...) in the validation loop.

Environment

• CUDA:
  - GPU: TITAN Xp
  - available: True
  - version: 10.2
• Packages:
  - numpy: 1.19.2
  - pyTorch_debug: False
  - pyTorch_version: 1.6.0
  - pytorch-lightning: 0.9.1rc4
  - tqdm: 4.49.0
• System:
  - OS: Linux
  - architecture: 64bit, ELF
  - processor: x86_64
  - python: 3.8.5
  - version: Typo: says seq-2seq in README #51~18.04.1-Ubuntu SMP Sat Sep 5 14:35:50 UTC 2020

Edit: Added reference to draft PR