Only track dev debugger events if enabled

[carmocca]

What does this PR do?

Noticed memory was building up from this while benchmarking the logger connector changes

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• [n/a] Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

[codecov]

Codecov Report

Merging #7875 (0b4e72d) into master (ce97676) will decrease coverage by 0%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #7875   +/-   ##
======================================
- Coverage      93%     92%   -0%     
======================================
  Files         202     202           
  Lines       13123   13125    +2     
======================================
- Hits        12156   12120   -36     
- Misses        967    1005   +38     

[ananthsub]

is this debugger used only for tests? should we look at removing this dependency and rebuilding event tracking later on?

[tchaton]

Not even. It was used in the past for debugging, but we found out it was risky to rely on it for testing.
I am not sure of its utility right now and could be deprecated.

[carmocca]

These are all the usages of the dev debugger in tests.

tests/callbacks/test_early_stopping.py:111:    assert len(trainer.dev_debugger.early_stopping_history) == expected_count
tests/callbacks/test_stochastic_weight_avg.py:100:            assert trainer.dev_debugger.count_events(
tests/plugins/test_rpc_sequential_plugin.py:46:        assert len(trainer.dev_debugger.pbar_added_metrics) > 0
tests/plugins/test_rpc_sequential_plugin.py:91:        assert len(trainer.dev_debugger.pbar_added_metrics) > 0
tests/models/test_amp.py:216:    assert trainer.dev_debugger.count_events('AMP') == 0
tests/models/test_amp.py:249:    assert trainer.dev_debugger.count_events('AMP') == 10
tests/checkpointing/test_model_checkpoint.py:143:    lr_scheduler_debug = trainer.dev_debugger.saved_lr_scheduler_updates
tests/checkpointing/test_model_checkpoint.py:249:    lr_scheduler_debug = trainer.dev_debugger.saved_lr_scheduler_updates
tests/checkpointing/test_model_checkpoint.py:867:    assert len(trainer.dev_debugger.checkpoint_callback_history) == 3
tests/checkpointing/test_model_checkpoint.py:954:    assert trainer.dev_debugger.checkpoint_callback_history[-1]['epoch'] == len(monitor) - 1
tests/checkpointing/test_checkpoint_callback_frequency.py:38:    assert len(trainer.dev_debugger.checkpoint_callback_history) == 0
tests/checkpointing/test_checkpoint_callback_frequency.py:47:    assert len(trainer.dev_debugger.checkpoint_callback_history) == 0
tests/trainer/test_dataloaders.py:527:        assert len(trainer.dev_debugger.num_seen_val_check_batches) == num_val_dataloaders
tests/trainer/test_dataloaders.py:528:        for dataloader_idx, num_batches in trainer.dev_debugger.num_seen_val_check_batches.items():
tests/trainer/test_dataloaders.py:532:        assert len(trainer.dev_debugger.num_seen_test_check_batches) == num_test_dataloaders
tests/trainer/test_dataloaders.py:533:        for dataloader_idx, num_batches in trainer.dev_debugger.num_seen_test_check_batches.items():
tests/trainer/test_dataloaders.py:594:        assert trainer.dev_debugger.num_seen_sanity_check_batches == trainer.num_sanity_val_steps * num_val_dataloaders
tests/trainer/test_dataloaders.py:1282:    assert len(trainer.dev_debugger.val_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1283:    assert len(trainer.dev_debugger.test_dataloader_calls) == 0
tests/trainer/test_dataloaders.py:1284:    assert len(trainer.dev_debugger.train_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1287:    calls = trainer.dev_debugger.dataloader_sequence_calls
tests/trainer/test_dataloaders.py:1315:    assert len(trainer.dev_debugger.val_dataloader_calls) == 10
tests/trainer/test_dataloaders.py:1316:    assert len(trainer.dev_debugger.test_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1317:    assert len(trainer.dev_debugger.train_dataloader_calls) == 3
tests/trainer/test_dataloaders.py:1320:    calls = trainer.dev_debugger.dataloader_sequence_calls
tests/trainer/test_dataloaders.py:1357:    assert len(trainer.dev_debugger.val_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1358:    assert len(trainer.dev_debugger.test_dataloader_calls) == 0
tests/trainer/test_dataloaders.py:1359:    assert len(trainer.dev_debugger.train_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1362:    calls = trainer.dev_debugger.dataloader_sequence_calls
tests/trainer/test_dataloaders.py:1389:    assert len(trainer.dev_debugger.val_dataloader_calls) == 4
tests/trainer/test_dataloaders.py:1390:    assert len(trainer.dev_debugger.train_dataloader_calls) == 3
tests/trainer/test_dataloaders.py:1391:    assert len(trainer.dev_debugger.test_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1394:    calls = trainer.dev_debugger.dataloader_sequence_calls
tests/trainer/test_dataloaders.py:1438:    assert len(trainer.dev_debugger.val_dataloader_calls) == 4
tests/trainer/test_dataloaders.py:1439:    assert len(trainer.dev_debugger.train_dataloader_calls) == 3
tests/trainer/test_dataloaders.py:1440:    assert len(trainer.dev_debugger.test_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1443:    calls = trainer.dev_debugger.dataloader_sequence_calls
tests/trainer/test_dataloaders.py:1490:    assert len(trainer.dev_debugger.val_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1491:    assert len(trainer.dev_debugger.test_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1492:    assert len(trainer.dev_debugger.train_dataloader_calls) == 1
tests/trainer/test_dataloaders.py:1495:    calls = trainer.dev_debugger.dataloader_sequence_calls
tests/trainer/flags/test_fast_dev_run.py:102:        assert len(trainer.dev_debugger.checkpoint_callback_history) == 0
tests/trainer/flags/test_fast_dev_run.py:106:        assert len(trainer.dev_debugger.early_stopping_history) == 0
tests/trainer/optimization/test_manual_optimization.py:98:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * num_manual_backward_calls
tests/trainer/optimization/test_manual_optimization.py:130:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * num_manual_backward_calls
tests/trainer/optimization/test_manual_optimization.py:161:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * num_manual_backward_calls
tests/trainer/optimization/test_manual_optimization.py:187:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * num_manual_backward_calls
tests/trainer/optimization/test_manual_optimization.py:225:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * num_manual_backward_calls
tests/trainer/optimization/test_manual_optimization.py:502:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * num_manual_backward_calls
tests/trainer/optimization/test_manual_optimization.py:577:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * 2
tests/trainer/optimization/test_manual_optimization.py:636:    assert trainer.dev_debugger.count_events('backward_call') == limit_train_batches * 2

Most could be replaced with progress tracking. Others could have the test changed to track manually.

[awaelchli]

agree, dev debugger should not exist at all. we want to rely on the existing built in, well tested unit testing framework.

[tchaton]

LGTM !